Introduction
In this lab, you will learn how to effectively use Gobuster, a popular directory and file brute-forcing tool, when a target web server is protected by Basic HTTP Authentication. Many web applications and administrative interfaces use Basic HTTP Authentication to restrict access to certain resources. Without providing valid credentials, Gobuster would typically be unable to discover these protected paths. This lab will guide you through the process of identifying such targets, understanding how to provide the necessary username and password to Gobuster, and successfully enumerating protected directories and files. By the end of this lab, you will be proficient in using Gobuster's authentication flags to bypass basic access restrictions and uncover hidden web content.
Identify a Target Protected by Basic HTTP Auth
In this step, you will identify a web server directory that is protected by Basic HTTP Authentication. We have set up a local Apache web server with a protected directory /protected_area. When you try to access this directory without credentials, the server will prompt for a username and password.
First, let's try to access the protected directory using curl without providing any credentials. This will demonstrate the authentication challenge.
Open your terminal and execute the following command:
curl http://localhost/protected_area/
You should see an output similar to this, indicating a 401 Unauthorized response and a WWW-Authenticate header:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
<hr>
<address>Apache/2.4.52 (Ubuntu) Server at localhost Port 80</address>
</body></html>
This output confirms that the /protected_area is protected by Basic HTTP Authentication.
Next, let's try to access a file inside this protected directory, secret.txt, to confirm it's also protected.
curl http://localhost/protected_area/secret.txt
You will receive a similar 401 Unauthorized response. This confirms that any resource within /protected_area requires authentication.
Obtain the Username and Password
In a real-world scenario, obtaining the username and password for Basic HTTP Authentication might involve various techniques such as social engineering, phishing, or brute-forcing. For the purpose of this lab, we have pre-configured the credentials for the protected area.
The username is labexuser and the password is labexpassword.
You can verify these credentials by using curl with the -u flag, which allows you to provide the username and password directly.
Execute the following command to access the protected directory with the correct credentials:
curl -u labexuser:labexpassword http://localhost/protected_area/
You should now see a successful response, likely an empty directory listing or a default index page if one existed. Since there's no index.html in /protected_area, you might see a 403 Forbidden or a directory listing if Indexes are enabled. The key is that you did not receive a 401 Unauthorized error.
Now, try to access the secret.txt file within the protected directory using the credentials:
curl -u labexuser:labexpassword http://localhost/protected_area/secret.txt
You should now see the content of the secret.txt file:
This is a secret file.
This confirms that the provided username and password are correct and grant access to the protected resources.
Use the -U Flag for the Username
In this step, you will learn how to specify the username for Basic HTTP Authentication in Gobuster using the -U flag. This flag is crucial when you know the username but might be brute-forcing the password or simply need to provide the username as part of the authentication process.
First, let's attempt a Gobuster scan without any authentication to see its behavior against the protected directory. We'll use a small wordlist for demonstration.
Create a simple wordlist file named common.txt in your ~/project directory:
echo -e "admin\nuser\nprotected_area\nsecret.txt" > ~/project/common.txt
Now, run Gobuster against http://localhost using this wordlist, but without authentication:
gobuster dir -u http://localhost -w ~/project/common.txt
You will notice that Gobuster will likely report 401 Unauthorized for the /protected_area entry, indicating it cannot access it without credentials.
/protected_area (Status: 401) [Size: 399]
Now, let's use the -U flag to specify the username labexuser. We will still omit the password for now to demonstrate the effect of just the username.
gobuster dir -u http://localhost -w ~/project/common.txt -U labexuser
Even with the username provided, Gobuster will still receive a 401 Unauthorized response because the password is also required. This step primarily demonstrates the syntax for providing the username.
/protected_area (Status: 401) [Size: 399]
This shows that while the username is accepted, the authentication still fails without the correct password.
Use the -P Flag for the Password
In this step, you will learn how to specify the password for Basic HTTP Authentication in Gobuster using the -P flag. This flag, combined with the -U flag, allows Gobuster to perform authenticated scans.
Continuing from the previous step, we know the username is labexuser and the password is labexpassword. Now, let's add the -P flag to provide the password.
Execute the following Gobuster command, including both the username (-U) and password (-P) flags:
gobuster dir -u http://localhost -w ~/project/common.txt -U labexuser -P labexpassword
Observe the output carefully. This time, when Gobuster encounters /protected_area, it should be able to authenticate successfully and report a 200 OK or 403 Forbidden status (depending on directory listing settings), rather than 401 Unauthorized.
/protected_area (Status: 403) [Size: 277]
The 403 Forbidden status indicates that access was granted (authentication successful), but the server is configured to forbid directory listings. This is a successful authentication, as opposed to the 401 Unauthorized we saw earlier.
This demonstrates that by providing both the username and password, Gobuster can successfully authenticate against Basic HTTP Authentication and proceed with its directory brute-forcing.
Execute the Scan and Access Protected Resources
In this final step, you will execute a full Gobuster scan against the protected area, ensuring that it can discover the secret.txt file. This demonstrates the practical application of using Basic HTTP Authentication with Gobuster to uncover hidden resources.
We will modify the target URL to specifically scan within the /protected_area and use a wordlist that includes secret.txt.
First, let's ensure our common.txt wordlist contains secret.txt. If you created it in Step 3, it should already be there.
Now, run Gobuster targeting http://localhost/protected_area/ with the correct username and password:
gobuster dir -u http://localhost/protected_area/ -w ~/project/common.txt -U labexuser -P labexpassword
After the scan completes, you should see an output similar to this, indicating that secret.txt was found with a 200 OK status:
/secret.txt (Status: 200) [Size: 21]
This confirms that Gobuster successfully authenticated and discovered the secret.txt file within the protected directory.
You can now verify the content of the discovered file using curl with the credentials:
curl -u labexuser:labexpassword http://localhost/protected_area/secret.txt
You should see:
This is a secret file.
This concludes the lab. You have successfully learned how to use Gobuster with Basic HTTP Authentication to discover protected web resources.
Summary
In this lab, you have gained practical experience in using Gobuster to enumerate directories and files on web servers protected by Basic HTTP Authentication. You started by identifying a protected target and understanding the 401 Unauthorized response. Then, you learned how to provide the necessary username and password to Gobuster using the -U and -P flags. Finally, you successfully executed an authenticated scan, demonstrating Gobuster's ability to discover hidden resources like secret.txt within a protected area. This skill is crucial for penetration testers and security professionals when dealing with web applications that employ basic authentication mechanisms.