Introduction
Gobuster is a powerful tool used for directory and file brute-forcing, DNS subdomain brute-forcing, and S3 bucket enumeration. When performing fuzzing operations, especially against web servers, it's common to encounter a large number of "noisy" results. These often include responses for non-existent paths that return a consistent HTTP status code (e.g., 404 Not Found) and, crucially, a consistent response body length. This can clutter the output and make it difficult to identify legitimate findings.
In this lab, you will learn how to leverage Gobuster's --exclude-length flag in fuzz mode. This feature allows you to specify one or more response body lengths to ignore, effectively filtering out these noisy results. By the end of this lab, you will be able to perform more targeted and efficient fuzzing scans, focusing only on the relevant responses.
Run a fuzz scan and identify a common response size for invalid requests
In this step, you will perform an initial Gobuster fuzz scan against a local web server. This will help you observe the typical output, especially the response lengths for non-existent paths, which often represent "noisy" results.
First, ensure the web server is running. You can check if the process is active:
ps aux | grep "python3 -m http.server 8000" | grep -v grep
You should see an output similar to this, indicating the server is running:
labex 1234 0.0 0.0 12345 6789 ? Sl HH:MM 0:00 python3 -m http.server 8000 --directory /tmp/web_root
Now, run a Gobuster fuzz scan using the wordlist created in the setup. We will target http://127.0.0.1:8000.
gobuster fuzz -u http://127.0.0.1:8000/FUZZ -w /tmp/wordlist.txt
Observe the output. You will notice several entries with the same status code (e.g., 404) and, more importantly, the same length. This consistent length for invalid requests is what we will target for exclusion. For example, nonexistentpath123 and anothernonexistentpath should show the same length.
/index.html (Status: 200) [Size: 19]
/admin (Status: 404) [Size: 19]
/login (Status: 404) [Size: 19]
/config (Status: 404) [Size: 19]
/robots.txt (Status: 404) [Size: 19]
/nonexistentpath123 (Status: 404) [Size: 19]
/anothernonexistentpath (Status: 404) [Size: 19]
From the output, identify the common response size for the 404 (Not Found) errors. In this example, it's 19. This is the length we will use to filter out the noise in the next step.
Rerun the scan using the --exclude-length flag with the identified size
In this step, you will rerun the Gobuster scan, but this time you will use the --exclude-length flag to filter out the noisy results based on the response length you identified in the previous step.
Recall the common response length for the 404 errors from the previous step. In our example, this was 19. Now, append --exclude-length 19 to your Gobuster command:
gobuster fuzz -u http://127.0.0.1:8000/FUZZ -w /tmp/wordlist.txt --exclude-length 19
Execute the command and observe the output.
Observe how the noisy results are filtered out
After running the command in the previous step, you should immediately notice a significant difference in the output. The entries corresponding to the excluded length (e.g., 404 responses with a length of 19) should no longer appear.
Compare the output from this scan with the output from Step 1.
Expected output:
/index.html (Status: 200) [Size: 19]
You should only see the /index.html entry, as it's the only one that returned a 200 OK status with a different content length (though in this specific setup, the 404 page also has a length of 19, which is a limitation of our simple server. In a real-world scenario, 404 pages often have a distinct length from valid pages). The key takeaway is that any response with the specified length is now filtered out.
This demonstrates how effectively the --exclude-length flag can reduce noise, allowing you to focus on potentially interesting findings.
Refine the exclusion with a comma-separated list of lengths
Sometimes, a web server might return different response lengths for various types of "not found" or "invalid" requests. Gobuster allows you to exclude multiple lengths by providing a comma-separated list to the --exclude-length flag.
To demonstrate this, let's imagine our server could return 404 pages with lengths of 19 and 50. We will simulate this by adding another "valid" path that returns a different length.
First, let's add a new file to our web root with a different size:
echo "This is a test page with a different length." > /tmp/web_root/testpage.html
Now, let's add testpage.html to our wordlist:
echo "testpage.html" >> /tmp/wordlist.txt
Run the scan again, but this time, let's assume we want to exclude both 19 and 39 (the length of testpage.html is 39).
gobuster fuzz -u http://127.0.0.1:8000/FUZZ -w /tmp/wordlist.txt --exclude-length 19,39
Observe the output. You should see that both the original 404 responses (length 19) and the testpage.html response (length 39) are now excluded.
Analyze the cleaner output for interesting findings
In this final step, you will analyze the cleaner output produced by using the --exclude-length flag. The goal is to understand how this filtering helps in identifying truly interesting findings during a fuzzing operation.
After running the command in the previous step, the output should be even more concise. If there were any other paths in your wordlist that returned a status code and length not in your exclusion list, they would now stand out.
For instance, if you had a path like /secret_admin_panel that returned a 200 OK with a unique length, it would be clearly visible in the filtered output, whereas it might have been lost among hundreds of 404s in an unfiltered scan.
In our current setup, with 19 and 39 excluded, and index.html having a length of 19, the output should be empty, as all entries are now excluded. This demonstrates the power of precise filtering.
This technique is invaluable in real-world penetration testing and bug bounty hunting, where large wordlists can generate overwhelming amounts of irrelevant data. By systematically excluding common "noise" lengths, you can significantly reduce the manual effort required to review scan results and increase your chances of discovering hidden directories or files.
To clean up the environment, you can stop the Python web server:
kill $(cat /tmp/web_server_pid)
This concludes the lab. You have successfully learned how to use Gobuster's --exclude-length flag to refine your fuzzing scans.
Summary
In this lab, you have gained practical experience in using Gobuster's --exclude-length flag in fuzz mode. You started by performing an initial scan to identify common response lengths for invalid requests, which often represent noise. Then, you learned how to use the --exclude-length flag with a single length to filter out these irrelevant results, leading to a cleaner output. Finally, you explored how to refine the exclusion further by providing a comma-separated list of lengths, demonstrating its flexibility in handling various types of noisy responses.
By mastering this technique, you can significantly improve the efficiency and effectiveness of your web application fuzzing operations, allowing you to focus on truly interesting findings and accelerate your vulnerability discovery process.