Introduction
In this lab, you will learn how to effectively use Gobuster for DNS subdomain enumeration, with a particular focus on handling wildcard DNS records. Wildcard DNS records can often lead to a large number of false positives in subdomain scans, making it difficult to identify legitimate subdomains. Gobuster provides a --wildcard flag that helps in filtering out these responses, ensuring cleaner and more accurate results. You will identify a domain with a wildcard record, observe the impact of a standard scan, and then use the --wildcard flag to see the difference.
Identify a Domain with a Wildcard DNS Record
In this step, you will identify a domain that uses a wildcard DNS record. A wildcard DNS record (*.example.com) means that any subdomain that doesn't have a specific DNS record will resolve to a predefined IP address. This can cause issues during subdomain enumeration as many non-existent subdomains will appear to resolve, leading to false positives.
We will use dig to query a non-existent subdomain for our target domain wildcard.labex.io to confirm its wildcard behavior.
First, let's check the content of our target domain file:
cat ~/project/target_domain.txt
You should see wildcard.labex.io as the output.
Now, let's query a non-existent subdomain, for example, nonexistent.wildcard.labex.io, using dig. If it resolves to an IP address, it indicates a wildcard record is in place.
dig nonexistent.wildcard.labex.io
Observe the ANSWER SECTION in the output. If you see an IP address for nonexistent.wildcard.labex.io, it confirms the presence of a wildcard DNS record.
; <<>> DiG 9.18.1-1ubuntu1.7-Ubuntu <<>> nonexistent.wildcard.labex.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12345
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nonexistent.wildcard.labex.io. IN A
;; ANSWER SECTION:
nonexistent.wildcard.labex.io. 300 IN A 192.0.2.123 ## Example IP, will vary
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Jan 01 12:00:00 UTC 2023
;; MSG SIZE rcvd: 78
The presence of an ANSWER SECTION for a clearly non-existent subdomain confirms that wildcard.labex.io has a wildcard DNS record.
Run a Standard DNS Scan and Observe False Positives
In this step, you will perform a standard DNS subdomain enumeration scan on wildcard.labex.io using Gobuster without any wildcard filtering. This will demonstrate how wildcard DNS records can lead to numerous false positives, making it difficult to distinguish between legitimate subdomains and those resolved by the wildcard.
We will use the gobuster dns command with the -d flag for the domain and the -w flag for our wordlist.
gobuster dns -d wildcard.labex.io -w ~/project/wordlist.txt
Observe the output. You will likely see that almost every entry from your wordlist resolves to an IP address, even if it's not a "real" subdomain. This is because the wildcard DNS record catches all these queries and resolves them.
Gobuster v3.6
by OJ <oj@zerokopter.com>
[+] Url: wildcard.labex.io
[+] Threads: 10
[+] Wordlist: /home/labex/project/wordlist.txt
[+] Timeout: 10s
[+] User Agent: gobuster/3.6
2023/01/01 12:00:00 Starting gobuster
Found: test.wildcard.labex.io (192.0.2.123)
Found: dev.wildcard.labex.io (192.0.2.123)
Found: www.wildcard.labex.io (192.0.2.123)
Found: mail.wildcard.labex.io (192.0.2.123)
Found: blog.wildcard.labex.io (192.0.2.123)
Found: admin.wildcard.labex.io (192.0.2.123)
Found: api.wildcard.labex.io (192.0.2.123)
Found: cdn.wildcard.labex.io (192.0.2.123)
Found: ftp.wildcard.labex.io (192.0.2.123)
Found: vpn.wildcard.labex.io (192.0.2.123)
2023/01/01 12:00:05 Finished
As you can see, all entries from the wordlist were "found," which is misleading due to the wildcard DNS record. This highlights the problem of false positives in subdomain enumeration.
Rerun the Scan with the --wildcard Flag
In this step, you will rerun the Gobuster DNS scan, but this time you will include the --wildcard flag. This flag tells Gobuster to perform an initial check for wildcard DNS records and then filter out any subsequent results that resolve to the same IP address as the wildcard. This significantly reduces false positives and provides a much cleaner output.
Execute the Gobuster command again, adding the --wildcard flag:
gobuster dns -d wildcard.labex.io -w ~/project/wordlist.txt --wildcard
Pay close attention to the output. You should notice that Gobuster first identifies the wildcard IP and then proceeds to filter out results that match it.
Gobuster v3.6
by OJ <oj@zerokopter.com>
[+] Url: wildcard.labex.io
[+] Threads: 10
[+] Wordlist: /home/labex/project/wordlist.txt
[+] Timeout: 10s
[+] User Agent: gobuster/3.6
[+] Wildcard DNS detected. Filtering responses for 192.0.2.123 ## Example IP
2023/01/01 12:00:00 Starting gobuster
2023/01/01 12:00:05 Finished
Notice the line [+] Wildcard DNS detected. Filtering responses for 192.0.2.123. This indicates that Gobuster successfully identified the wildcard IP and is now actively filtering out results that resolve to it. In this specific example, since all subdomains resolve to the wildcard IP, you might see no "Found" entries, which is the correct behavior for a domain with a full wildcard setup and no legitimate subdomains in your wordlist.
Observe How Gobuster Filters Out Wildcard Responses
In this step, we will delve deeper into how Gobuster's --wildcard flag works by observing its behavior with a slightly modified scenario. While our previous example showed a complete filtering, it's important to understand the mechanism. Gobuster first queries a random, non-existent subdomain (e.g., randomstring.yourdomain.com). If this query resolves to an IP address, Gobuster considers that IP address to be the wildcard IP. Any subsequent subdomain found during the enumeration that resolves to this same wildcard IP will be filtered out. Only subdomains resolving to a different IP address will be reported.
To illustrate this, let's imagine our wildcard.labex.io domain also had a legitimate www.wildcard.labex.io subdomain pointing to a different IP address. If that were the case, Gobuster would still filter out all the other wildcard entries but would report www.wildcard.labex.io.
For the purpose of this lab, since wildcard.labex.io is configured to have all subdomains resolve to the same wildcard IP, the output from the previous step (where no subdomains were "found") is the expected and correct behavior, demonstrating effective filtering.
Let's re-run the command and focus on the initial output line that confirms wildcard detection:
gobuster dns -d wildcard.labex.io -w ~/project/wordlist.txt --wildcard
The key line to observe is: [+] Wildcard DNS detected. Filtering responses for <IP_ADDRESS>. This confirms that Gobuster has identified the wildcard IP and is actively using it to filter results.
Gobuster v3.6
by OJ <oj@zerokopter.com>
[+] Url: wildcard.labex.io
[+] Threads: 10
[+] Wordlist: /home/labex/project/wordlist.txt
[+] Timeout: 10s
[+] User Agent: gobuster/3.6
[+] Wildcard DNS detected. Filtering responses for 192.0.2.123 ## Example IP
2023/01/01 12:00:00 Starting gobuster
2023/01/01 12:00:05 Finished
This step reinforces the understanding that Gobuster's --wildcard flag is crucial for obtaining accurate subdomain enumeration results on domains with wildcard DNS records.
Analyze the Cleaner, More Accurate Results
In this final step, you will analyze the difference between the results obtained from the standard Gobuster scan and the scan performed with the --wildcard flag. The goal is to appreciate how the --wildcard flag provides cleaner and more accurate results by eliminating false positives.
Recall the output from Step 2 (without --wildcard):
Every entry from the wordlist was reported as "Found," even though they were just resolving due to the wildcard DNS record. This output is noisy and makes it impossible to identify truly existing subdomains.
Recall the output from Step 3 and 4 (with --wildcard):
Gobuster explicitly stated [+] Wildcard DNS detected. Filtering responses for <IP_ADDRESS>. In our specific case, since all subdomains resolve to the wildcard IP, no "Found" entries were reported. This is the correct and accurate result, indicating that none of the subdomains in our wordlist are legitimate and resolve to a unique IP address.
If there were legitimate subdomains (e.g., www.wildcard.labex.io resolving to a different IP than the wildcard), they would have been listed in the output when using the --wildcard flag. This demonstrates the power of the flag in distinguishing real subdomains from wildcard entries.
The --wildcard flag is essential for efficient and accurate subdomain enumeration, especially when dealing with large wordlists and domains that extensively use wildcard DNS records. It saves time by reducing the need for manual filtering of false positives and ensures that your enumeration efforts are focused on truly relevant subdomains.
You have successfully learned how to identify wildcard DNS records and use Gobuster's --wildcard flag to obtain accurate subdomain enumeration results.
Summary
In this lab, you have gained practical experience in handling wildcard DNS records during subdomain enumeration using Gobuster. You learned to identify domains with wildcard records, observed how standard scans produce false positives, and most importantly, mastered the use of the --wildcard flag to filter out these misleading results. This skill is crucial for anyone performing reconnaissance or penetration testing, as it ensures the accuracy and efficiency of your subdomain discovery process. By applying the --wildcard flag, you can focus on legitimate subdomains, saving valuable time and resources.