Set a Custom User-Agent for Scans in Gobuster

Introduction

In web security, reconnaissance is a crucial first step. Tools like Gobuster are widely used for directory and file enumeration. However, many web servers and Web Application Firewalls (WAFs) employ simple detection mechanisms, such as blocking requests with suspicious User-Agent strings. By default, Gobuster uses a User-Agent string that can be easily identified.

In this lab, you will learn how to set a custom User-Agent string for your Gobuster scans. This technique can help you evade simple detection mechanisms and make your reconnaissance efforts more stealthy, allowing you to potentially discover more hidden directories and files.

Understand the Purpose of a User-Agent String

In this step, you will learn what a User-Agent string is and why it's important in web requests.

The User-Agent string is a header sent by a client (like a web browser or a tool like Gobuster) to a web server. It typically contains information about the client's application type, operating system, software vendor, and software version. Web servers use this information to deliver content optimized for the client or to log client statistics.

For security tools like Gobuster, the default User-Agent string often reveals the tool's identity, which can trigger WAFs or intrusion detection systems (IDS). For example, Gobuster's default User-Agent might look something like gobuster/3.1.0.

Let's perform a basic Gobuster scan without any custom User-Agent to see its default behavior. We will scan a local web server that was set up in the background.

First, ensure the web server is running by checking its process:

ps aux | grep "python3 -m http.server 8080" | grep -v grep

You should see an output similar to this, indicating the server is running:

labex      1234  0.0  0.1  12345  6789 ?        Sl   HH:MM   0:00 python3 -m http.server 8080 --directory /tmp/web_root

Now, run a simple Gobuster scan against the local server on port 8080. We'll use a small wordlist for demonstration.

gobuster dir -u http://127.0.0.1:8080 -w /usr/share/wordlists/dirb/common.txt -q -x html,txt -t 10

• -u http://127.0.0.1:8080: Specifies the target URL.
• -w /usr/share/wordlists/dirb/common.txt: Specifies the wordlist to use.
• -q: Quiet mode, only prints results.
• -x html,txt: Specifies extensions to look for.
• -t 10: Sets the number of concurrent threads to 10.

You will see output similar to this, showing discovered directories and files:

/admin                (Status: 200)
/secret               (Status: 200)
/backup               (Status: 200)

While this scan works, the User-Agent used by Gobuster is its default one, which could be easily blocked by a WAF.

Find a Common Browser User-Agent String

In this step, you will learn how to find a common browser User-Agent string that you can use to masquerade your Gobuster scans.

To make your Gobuster scans appear as if they are coming from a legitimate web browser, you need to use a User-Agent string that a typical browser would send. You can find these strings by:

1. Inspecting network requests in your browser's developer tools: When you browse a website, open your browser's developer tools (usually F12), go to the "Network" tab, and inspect the headers of any request.
2. Searching online: Many websites compile lists of common User-Agent strings.

For this lab, let's use a common User-Agent string for a recent version of Chrome on Windows. A typical User-Agent string for Chrome might look like this:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

This string indicates:

• Mozilla/5.0: A general token that many browsers include for historical reasons.
• (Windows NT 10.0; Win64; x64): Operating system information (Windows 10, 64-bit).
• AppleWebKit/537.36 (KHTML, like Gecko): Rendering engine information (WebKit, with KHTML compatibility).
• Chrome/120.0.0.0: Browser name and version.
• Safari/537.36: Another historical token, often included by WebKit-based browsers.

You can copy this string directly for the next step. It's important to enclose the User-Agent string in quotes when using it in the command line, especially if it contains spaces or special characters.

Use the -a Flag to Set a Custom User-Agent

In this step, you will learn how to use the -a flag in Gobuster to set a custom User-Agent string for your scans.

Gobuster provides the -a flag (or --agent) specifically for this purpose. You simply pass your desired User-Agent string as an argument to this flag.

Let's prepare the Gobuster command with the custom User-Agent string we identified in the previous step. We will use the same target and wordlist.

CUSTOM_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
echo "Using custom User-Agent: $CUSTOM_USER_AGENT"

Now, construct the Gobuster command. Notice how the -a flag is used:

gobuster dir -u http://127.0.0.1:8080 -w /usr/share/wordlists/dirb/common.txt -a "$CUSTOM_USER_AGENT" -q -x html,txt -t 10

• -a "$CUSTOM_USER_AGENT": This is the new part, where we pass our custom User-Agent string. The quotes around $CUSTOM_USER_AGENT are crucial because the string contains spaces.

Before executing, consider the difference this makes. Without the -a flag, the web server would see a request from "gobuster". With the -a flag, it will see a request from "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", which looks like a legitimate browser.

Execute the Scan with the New User-Agent

In this step, you will execute the Gobuster scan using the custom User-Agent string and observe the results.

Now, run the Gobuster command with the custom User-Agent. The output of the scan itself will look similar to the previous one, as the User-Agent only affects how the request is perceived by the server, not the scan results themselves (unless the server blocks the default User-Agent).

Execute the command:

CUSTOM_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
gobuster dir -u http://127.0.0.1:8080 -w /usr/share/wordlists/dirb/common.txt -a "$CUSTOM_USER_AGENT" -q -x html,txt -t 10

You should see the same directories and files discovered:

/admin                (Status: 200)
/secret               (Status: 200)
/backup               (Status: 200)

Although the output is the same, the crucial difference is that if this were a real-world scenario with a WAF or IDS, this scan would be less likely to be flagged as suspicious compared to a scan using Gobuster's default User-Agent. This is a fundamental technique in ethical hacking and penetration testing to avoid detection.

Discuss Why This Can Help Evade Simple Filters

In this step, we will discuss why setting a custom User-Agent can be an effective technique for evading simple filters and detection mechanisms.

Many web application firewalls (WAFs), intrusion detection systems (IDS), and even basic server configurations employ rules to identify and block suspicious traffic. One of the simplest and most common rules is to look at the User-Agent string.

Here's why setting a custom User-Agent helps:

• Bypassing User-Agent based Blocking: If a WAF has a rule to block requests with User-Agents like "gobuster", "nmap", "sqlmap", or other known security tools, using a common browser User-Agent will allow your requests to pass through undetected by this specific rule.
• Reducing Suspicion: A server administrator reviewing logs might notice a high volume of requests from an unusual User-Agent (like gobuster/3.1.0). By mimicking a common browser, your requests blend in with legitimate user traffic, making them less likely to be manually investigated or automatically flagged.
• Evading Rate Limiting (in some cases): While not directly related to User-Agent, some systems might apply stricter rate limits to non-browser User-Agents. By appearing as a browser, you might avoid these stricter limits, allowing your scan to proceed more quickly or completely.

It's important to note that this technique is effective against simple filters. More sophisticated WAFs and IDSs use a combination of factors (e.g., request patterns, IP reputation, behavioral analysis) to detect malicious activity, so a custom User-Agent alone might not be sufficient to bypass advanced defenses. However, it's a fundamental and often necessary first step in stealthy reconnaissance.

To clean up the environment, you can stop the Python web server:

pkill -f "python3 -m http.server 8080"

This command finds and terminates the process running the Python web server.

Summary

In this lab, you learned the importance of the User-Agent string in web requests and how it can be used by security tools like Gobuster. You successfully identified a common browser User-Agent string and, most importantly, used the -a flag in Gobuster to set a custom User-Agent for your scans. This practical skill is crucial for making your reconnaissance efforts more stealthy and effective, helping you to bypass simple detection mechanisms and blend in with legitimate web traffic.