Introduction
In this lab, you will learn a powerful technique for target discovery by combining the capabilities of Nmap and Gobuster. Nmap is a versatile network scanner used to discover hosts and services on a computer network, while Gobuster is a tool used to brute-force URIs (directories and files) and DNS subdomains. By integrating these two tools, you can automate the process of identifying potential web servers within a given network range and then systematically discover hidden directories and files on those servers. This approach significantly enhances your reconnaissance efforts, making the discovery process more efficient and comprehensive.
Use Nmap to Scan a Network Range for Open Web Ports (80, 443)
In this step, you will use Nmap to scan a local network range for hosts that have common web ports (80 for HTTP and 443 for HTTPS) open. This is the initial phase of identifying potential web servers. We will scan the 172.17.0.0/24 network range, which is typically used by Docker for its internal networks.
First, let's ensure Nmap is installed. It should be pre-installed on the LabEx VM.
nmap --version
You should see output similar to this, indicating Nmap is installed:
Nmap version 7.80 ( https://nmap.org )
Platform: x86_64-pc-linux-gnu
Compiled with: liblua-5.3.3 openssl-1.1.1f libpcre-8.39 libpcap-1.9.1 nmap-libdnet-1.12 ipv6
Compiled without:
Available nsock engines: epoll poll select
Now, execute the Nmap scan. We will use the -p flag to specify ports 80 and 443, and -oG to output the results in Grepable format, which is easier to parse.
nmap -p 80,443 -oG nmap_web_scan.txt 172.17.0.0/24
This command will scan the specified network range for hosts with ports 80 or 443 open and save the output to nmap_web_scan.txt. The scan might take a few moments to complete.
After the scan finishes, you can view the content of the output file:
cat nmap_web_scan.txt
You will see lines indicating discovered hosts and their open ports. For example:
## Nmap 7.80 scan initiated Mon Jan 1 00:00:00 2024 as: nmap -p 80,443 -oG nmap_web_scan.txt 172.17.0.0/24
Host: 172.17.0.2 () Ports: 80/open/tcp//http///, 443/open/tcp//https///
Host: 172.17.0.3 () Ports: 80/open/tcp//http///
## Nmap done at Mon Jan 1 00:00:00 2024 -- 256 IP addresses (2 hosts up) scanned in 5.00 seconds
Parse the Nmap Output to Create a List of Web Server URLs
In this step, you will parse the nmap_web_scan.txt file to extract the IP addresses of hosts with open web ports and format them into a list of URLs. This list will then be used as input for Gobuster.
We will use grep and awk to process the nmap_web_scan.txt file. The goal is to extract lines containing "open" ports 80 or 443, then extract the IP address, and finally construct the URL (e.g., http://172.17.0.2 or https://172.17.0.2).
Execute the following command to parse the Nmap output and create web_servers.txt:
grep "open" nmap_web_scan.txt | awk '/80\/open/ {print "http://" $2} /443\/open/ {print "https://" $2}' | sort -u > web_servers.txt
Let's break down this command:
grep "open" nmap_web_scan.txt: Filters lines fromnmap_web_scan.txtthat contain the word "open", indicating an open port.awk '/80\/open/ {print "http://" $2} /443\/open/ {print "https://" $2}': Processes the filtered lines./80\/open/ {print "http://" $2}: If a line contains "80/open", it prints "http://" followed by the second field ($2), which is the IP address./443\/open/ {print "https://" $2}: If a line contains "443/open", it prints "https://" followed by the second field ($2), which is the IP address.- Note: A host might have both 80 and 443 open, resulting in two URLs for the same IP.
sort -u: Sorts the URLs and removes any duplicate entries.> web_servers.txt: Redirects the final output to a new file namedweb_servers.txt.
Now, view the content of the web_servers.txt file:
cat web_servers.txt
You should see a list of URLs, one per line, similar to this:
http://172.17.0.2
https://172.17.0.2
http://172.17.0.3
This file now contains the targets for our Gobuster scan.
Write a Simple Bash Script to Loop Through the URL List
In this step, you will create a Bash script that reads each URL from web_servers.txt and prepares for the Gobuster scan. This script will serve as the automation backbone for our combined Nmap and Gobuster workflow.
First, let's create a new script file named gobust_scan.sh using nano:
nano gobust_scan.sh
Paste the following content into the gobust_scan.sh file:
#!/bin/bash
## Define the wordlist for Gobuster
WORDLIST="/usr/share/wordlists/dirb/common.txt" ## A common wordlist for directory brute-forcing
## Check if the wordlist exists
if [ ! -f "$WORDLIST" ]; then
echo "Error: Wordlist not found at $WORDLIST. Please ensure it exists."
echo "You might need to install 'dirb' or 'wordlists' package, e.g., sudo apt install dirb"
exit 1
fi
## Check if web_servers.txt exists
if [ ! -f "web_servers.txt" ]; then
echo "Error: web_servers.txt not found. Please run Step 2 first."
exit 1
fi
echo "Starting Gobuster scans..."
echo "-------------------------"
## Loop through each URL in web_servers.txt
while IFS= read -r url; do
echo "Scanning $url..."
## Placeholder for Gobuster command - will be added in the next step
## gobuster dir -u "$url" -w "$WORDLIST" -o "gobuster_$(echo $url | sed -e 's/http:\/\///g' -e 's/https:\/\///g' -e 's/\//_/g').txt"
echo "Finished scanning $url."
echo "-------------------------"
done < web_servers.txt
echo "All scans completed."
Save the file by pressing Ctrl+X, then Y to confirm, and Enter to save to the current filename.
Next, make the script executable:
chmod +x gobust_scan.sh
Now, let's test the script to ensure it loops through the URLs correctly. It won't run Gobuster yet, but it will print messages for each URL.
./gobust_scan.sh
You should see output similar to this, indicating that the script is correctly reading and processing each URL:
Starting Gobuster scans...
-------------------------
Scanning http://172.17.0.2...
Finished scanning http://172.17.0.2.
-------------------------
Scanning https://172.17.0.2...
Finished scanning https://172.17.0.2.
-------------------------
Scanning http://172.17.0.3...
Finished scanning http://172.17.0.3.
-------------------------
All scans completed.
This confirms that our script is ready to incorporate the Gobuster command.
Call Gobuster within the Script for Each Discovered Target
In this step, you will modify the gobust_scan.sh script to integrate the Gobuster command. For each URL discovered by Nmap, Gobuster will attempt to find hidden directories and files using a common wordlist.
First, let's ensure Gobuster is installed. It should be pre-installed on the LabEx VM.
gobuster --version
You should see output similar to this:
gobuster v3.1.0
Now, open the gobust_scan.sh script again for editing:
nano gobust_scan.sh
Locate the placeholder line:
## gobuster dir -u "$url" -w "$WORDLIST" -o "gobuster_$(echo $url | sed -e 's/http:\/\///g' -e 's/https:\/\///g' -e 's/\//_/g').txt"
Uncomment this line and ensure it looks exactly like this:
#!/bin/bash
## Define the wordlist for Gobuster
WORDLIST="/usr/share/wordlists/dirb/common.txt" ## A common wordlist for directory brute-forcing
## Check if the wordlist exists
if [ ! -f "$WORDLIST" ]; then
echo "Error: Wordlist not found at $WORDLIST. Please ensure it exists."
echo "You might need to install 'dirb' or 'wordlists' package, e.g., sudo apt install dirb"
exit 1
fi
## Check if web_servers.txt exists
if [ ! -f "web_servers.txt" ]; then
echo "Error: web_servers.txt not found. Please run Step 2 first."
exit 1
fi
echo "Starting Gobuster scans..."
echo "-------------------------"
## Loop through each URL in web_servers.txt
while IFS= read -r url; do
echo "Scanning $url..."
## Gobuster command
gobuster dir -u "$url" -w "$WORDLIST" -o "gobuster_$(echo $url | sed -e 's/http:\/\///g' -e 's/https:\/\///g' -e 's/\//_/g').txt"
echo "Finished scanning $url."
echo "-------------------------"
done < web_servers.txt
echo "All scans completed."
Save the modified script (Ctrl+X, Y, Enter).
Now, execute the script. This will run Gobuster for each URL and save the results to separate files.
./gobust_scan.sh
The script will now execute Gobuster for each URL. You will see Gobuster's output directly in your terminal as it runs, and it will also save the results to files named like gobuster_172.17.0.2.txt, gobuster_172.17.0.2_443.txt, etc. (the exact filenames depend on the URLs).
After the script completes, you can list the generated Gobuster output files:
ls gobuster_*.txt
You should see files similar to:
gobuster_172.17.0.2.txt gobuster_172.17.0.2_443.txt gobuster_172.17.0.3.txt
Consolidate the Gobuster Results from All Targets
In this final step, you will consolidate all the individual Gobuster output files into a single, comprehensive report. This makes it easier to review all discovered directories and files across all scanned web servers.
We will use the cat command to concatenate all files matching gobuster_*.txt into a single file named all_gobuster_results.txt.
cat gobuster_*.txt > all_gobuster_results.txt
This command will take the content of all files starting with gobuster_ and ending with .txt and combine them into all_gobuster_results.txt.
Now, you can view the consolidated results:
cat all_gobuster_results.txt
You will see a combined output of all the directories and files Gobuster found on the scanned web servers. The output will vary based on the content of the web servers, but it might look something like this:
===============================================================
Gobuster v3.1.0
...
[+] Found: /index.html (Status: 200)
[+] Found: /css (Status: 301)
[+] Found: /js (Status: 301)
...
===============================================================
Gobuster v3.1.0
...
[+] Found: /admin (Status: 301)
[+] Found: /login.php (Status: 200)
...
This consolidated file provides a single point of reference for all discovered web assets, completing your automated target discovery process.
Summary
In this lab, you successfully combined Nmap and Gobuster to automate the process of web server and directory discovery. You started by using Nmap to scan a network range for open web ports, then parsed its output to create a list of target URLs. Next, you developed a Bash script to iterate through these URLs, calling Gobuster for each to find hidden directories and files. Finally, you consolidated all the Gobuster results into a single report, demonstrating an efficient and comprehensive reconnaissance workflow. This integrated approach is a valuable skill for anyone involved in network security or web application testing.