Filter Scan Results by Status Codes in Gobuster

Introduction

In web enumeration, tools like Gobuster are used to discover hidden directories and files on web servers. However, the raw output can be overwhelming, containing many entries that are not immediately relevant. Filtering these results based on HTTP status codes is a powerful technique to narrow down your focus to specific types of responses, such as successful pages (200 OK), redirects (301/302), or forbidden resources (403).

This lab will guide you through using Gobuster's filtering capabilities to efficiently analyze scan results. You will learn to identify common HTTP status codes, perform a basic Gobuster scan, and then apply filters to display only the status codes you are interested in. This skill is essential for anyone involved in web security or penetration testing, as it helps in quickly identifying actionable information.

Understand Common HTTP Status Codes (200, 301, 403)

In this step, you will learn about some of the most common HTTP status codes that are frequently encountered during web enumeration. Understanding these codes is fundamental to interpreting Gobuster's output and effectively filtering results.

• 200 OK: This status code indicates that the request has succeeded. The server has successfully processed the request, and the requested resource (e.g., a web page, an image) is being returned. In Gobuster scans, a 200 status code usually means a valid, accessible page or directory was found.
• 301 Moved Permanently: This status code signifies that the requested resource has been permanently moved to a new URL. The browser (or Gobuster) should automatically redirect to the new location. While not an error, finding 301 redirects can indicate old or reorganized content.
• 403 Forbidden: This status code means that the server understood the request but refuses to authorize it. This typically happens when a user (or Gobuster) tries to access a resource that requires authentication or specific permissions, but the client does not have them. Finding 403s can be interesting as they might point to sensitive areas that are protected but still exist.

To get a feel for these, you can try accessing some paths on our dummy web server using curl.

First, let's confirm the server is running by trying to access index.html:

curl -I http://localhost:8000/index.html

You should see a HTTP/1.0 200 OK status in the output.

Next, try accessing redirect.html to see a 301 redirect:

curl -I http://localhost:8000/redirect.html

You should observe a HTTP/1.0 301 Moved Permanently status.

Finally, attempt to access admin.html to see a 403 Forbidden response:

curl -I http://localhost:8000/admin.html

This command should return a HTTP/1.0 403 Forbidden status.

These curl commands help you manually verify how different paths on the server respond with specific HTTP status codes, which is what Gobuster will automate.

Run a Scan to See All Status Codes

In this step, you will perform a basic Gobuster scan against the dummy web server without any status code filtering. This will allow you to see the default output, which includes all discovered paths along with their corresponding HTTP status codes. This serves as a baseline before applying filters.

The basic syntax for Gobuster directory brute-forcing is gobuster dir -u <URL> -w <wordlist>.

• -u: Specifies the target URL. In our case, it's http://localhost:8000.
• -w: Specifies the path to the wordlist file. We will use the wordlist.txt file created in the setup.

Execute the following command in your terminal:

gobuster dir -u http://localhost:8000 -w ~/project/wordlist.txt

After running the command, observe the output. You should see lines similar to these, indicating the discovered paths and their status codes:

/index.html           (Status: 200)
/secret.html          (Status: 200)
/redirect.html        (Status: 301)
/admin.html           (Status: 403)

Notice that Gobuster reports all status codes it encounters. In a real-world scenario with a large wordlist, this output can be very long and contain many irrelevant entries. The next steps will show you how to refine this output.

Use the -s Flag to Show Only 200 OK and 301 Redirects

In this step, you will learn how to use the -s (status) flag in Gobuster to filter the scan results and display only specific HTTP status codes. This is incredibly useful when you are looking for particular types of responses, such as successful pages or redirects.

The -s flag allows you to specify a comma-separated list of status codes to include in the output. For example, to see only 200 (OK) and 301 (Moved Permanently) responses, you would use -s 200,301.

Execute the following command:

gobuster dir -u http://localhost:8000 -w ~/project/wordlist.txt -s 200,301

Observe the output. This time, you should only see entries with status codes 200 and 301:

/index.html           (Status: 200)
/secret.html          (Status: 200)
/redirect.html        (Status: 301)

Notice that the /admin.html entry, which returned a 403 status, is no longer present in the output. This demonstrates how effectively the -s flag can narrow down your results, making it easier to focus on relevant findings.

Run a New Scan to Find Forbidden Pages (403)

In this step, you will perform another filtered scan, this time specifically looking for "Forbidden" pages, which return a 403 HTTP status code. Identifying 403 responses can be crucial in penetration testing, as they often indicate protected administrative interfaces, sensitive files, or directories that might be vulnerable to misconfigurations or bypass techniques.

To filter for only 403 status codes, you will use the -s flag with 403 as the argument.

Execute the following command:

gobuster dir -u http://localhost:8000 -w ~/project/wordlist.txt -s 403

Examine the output. You should now only see the entry for /admin.html, which returned a 403 status:

/admin.html           (Status: 403)

This targeted scan quickly highlights resources that the server explicitly denies access to. While these resources are protected, their mere existence can be valuable information for further investigation. This demonstrates the flexibility of the -s flag in focusing on specific types of server responses.

Analyze the Filtered Output

In this final step, you will analyze the implications of the filtered outputs you've generated. Understanding what different status codes mean in the context of a Gobuster scan is key to effective web enumeration.

When you filtered for 200,301 status codes, you identified:

• /index.html (200 OK): This is the main page, indicating a successful and accessible resource.
• /secret.html (200 OK): This also returned 200, suggesting it's an accessible page. In a real scenario, you would investigate its content to see if it contains sensitive information.
• /redirect.html (301 Moved Permanently): This indicates a redirection. You would typically follow the redirect to see where it leads, as it might point to updated content or a different part of the application.

When you filtered for 403 status codes, you identified:

• /admin.html (403 Forbidden): This is a crucial finding. A 403 response means the server knows the resource exists but denies access. This often points to administrative panels, sensitive directories, or files that are intentionally protected. While direct access is denied, this path is a prime candidate for further investigation, such as:
  • Bypass attempts: Are there any known bypass techniques for this specific web server or application?
  • Default credentials: Could default credentials allow access?
  • Vulnerability scanning: Are there any vulnerabilities that could lead to unauthorized access?

By selectively filtering Gobuster's output, you can quickly pivot your investigation towards the most promising leads, saving time and making your enumeration process more efficient and targeted. This skill is fundamental for any security professional performing web application assessments.

Summary

In this lab, you have learned how to effectively use Gobuster's -s flag to filter scan results based on HTTP status codes. You started by understanding common status codes like 200 (OK), 301 (Moved Permanently), and 403 (Forbidden). You then performed a basic Gobuster scan to see all responses and subsequently applied filters to focus on specific types of results.

By filtering for 200 and 301 status codes, you identified accessible pages and redirects. More importantly, by filtering for 403 status codes, you pinpointed potentially sensitive or protected resources that warrant further investigation. This targeted approach significantly streamlines the web enumeration process, allowing you to quickly identify actionable intelligence during penetration testing or security assessments. Mastering this technique is a valuable skill for any cybersecurity professional.