Introduction
Curl is a powerful command-line tool that allows you to transfer data using various protocols, including HTTP, FTP, and SFTP. In this tutorial, you will learn how to use Curl to access different ports on your server or network, enabling you to troubleshoot connectivity issues and test port availability.
This hands-on lab will guide you through basic and advanced Curl commands for port access, helping you understand how network services operate on different ports. By the end of this lab, you will be able to confidently use Curl to interact with services running on various ports.
Getting Started with Curl
Curl, which stands for "Client URL," is a command-line tool for transferring data to or from a server. It supports numerous protocols including HTTP, HTTPS, FTP, SFTP, and many more. Before diving into port-specific operations, let's ensure you understand the basic usage of Curl.
Basic Curl Command
Open your terminal and type the following command to check if Curl is installed on your system:
curl --version
You should see output similar to this, showing the Curl version and supported features:
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd
Now, let's try a simple Curl request to a website. Type the following command:
curl https://example.com
This command downloads the HTML content of example.com and displays it in your terminal. The output will be similar to:
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<!-- More HTML content -->
</head>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents...</p>
<!-- More HTML content -->
</div>
</body>
</html>
Understanding HTTP Ports
Web servers typically run on specific ports:
- Port 80 for HTTP (unsecured)
- Port 443 for HTTPS (secured)
When you access a website without specifying a port, your browser automatically uses these default ports. However, with Curl, you can explicitly specify which port to connect to.
Let's try accessing a website on the standard HTTP port (port 80):
curl http://example.com:80
The output should be similar to your previous Curl command, as port 80 is the default for HTTP.
Now, try accessing the same website on the HTTPS port (port 443):
curl https://example.com:443
Again, the output should be similar, as port 443 is the default for HTTPS.
In these examples, we've explicitly specified the ports in the URL using the format protocol://domain:port. This syntax is crucial when working with services on non-standard ports.
Using Curl with Different Ports
Now that you understand the basics of Curl and port specification, let's explore how to interact with different ports.
Accessing Non-Standard Web Ports
Web servers can run on ports other than 80 and 443. For testing purposes, let's set up a simple web server on port 8000 using Python's built-in HTTP server.
First, create a simple HTML file to serve:
echo "<html><body><h1>Hello from port 8000!</h1></body></html>" > ~/project/test.html
Now, navigate to the directory containing the file and start a simple HTTP server on port 8000:
cd ~/project
python3 -m http.server 8000 &
The & at the end of the command runs the server in the background. You should see output like:
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
Now, use Curl to access this server on port 8000:
curl http://localhost:8000/test.html
You should see the HTML content we created:
<html>
<body>
<h1>Hello from port 8000!</h1>
</body>
</html>
When we're done with the server, we can stop it by finding its process ID and killing it:
ps aux | grep "python3 -m http.server"
This will show output similar to:
labex 1234 0.0 0.1 235368 12312 pts/0 S 10:00 0:00 python3 -m http.server 8000
labex 1235 0.0 0.0 12345 1234 pts/0 S+ 10:01 0:00 grep --color=auto python3 -m http.server
Note the process ID (PID) in the second column (1234 in this example), and kill the process:
kill $(pgrep -f "python3 -m http.server")
Using Different HTTP Methods
Curl allows you to specify different HTTP methods using the -X flag. Let's try a POST request:
curl -X POST http://example.com
This sends a POST request to example.com. Most websites will respond with a message indicating they expect different data for POST requests.
Adding Headers to Requests
When working with APIs or specific services, you often need to send custom headers. You can do this with the -H flag:
curl -H "Content-Type: application/json" http://example.com
This sends a request with a JSON content-type header. The response will depend on how the server handles this header.
Viewing Response Headers
To see the response headers along with the content, use the -i flag:
curl -i http://example.com
This will show output similar to:
HTTP/1.1 200 OK
Content-Encoding: gzip
Accept-Ranges: bytes
Age: 558039
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Wed, 07 Jun 2023 12:34:56 GMT
Etag: "3147526947+gzip"
Expires: Wed, 14 Jun 2023 12:34:56 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECS (dcb/7F5B)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 1256
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<!-- More HTML content -->
</head>
<!-- More HTML content -->
</html>
The headers provide valuable information about the server, content type, caching policies, and more.
Advanced Port Access with Curl
Now that you understand the basics of using Curl with different ports, let's explore more advanced techniques for port access and testing.
Testing Port Availability
Curl can be used to check if a specific port is open on a server. When a port is open, Curl will attempt to establish a connection and potentially receive data. When a port is closed, Curl will report an error.
Let's test some common ports:
## Test if port 80 (HTTP) is open
curl -s -o /dev/null -w "%{http_code}\n" http://example.com:80
This command will display the HTTP status code (typically 200 if the port is open and the server is responding correctly). The -s flag makes Curl silent, -o /dev/null redirects the output to nowhere, and -w "%{http_code}\n" prints just the HTTP status code.
Let's try a few more ports:
## Test if port 443 (HTTPS) is open
curl -s -o /dev/null -w "%{http_code}\n" https://example.com:443
This should return 200 or another HTTP status code, indicating that port 443 is open.
Now, let's test a port that is likely closed:
## Test if port 81 (uncommon) is open
curl -s --connect-timeout 5 http://example.com:81
This command will likely fail with an error message like:
curl: (7) Failed to connect to example.com port 81: Connection refused
or it might time out after 5 seconds (as specified by --connect-timeout):
curl: (28) Connection timed out after 5001 milliseconds
Accessing FTP Servers
Curl supports multiple protocols, including FTP. Let's see how to access an FTP server:
curl ftp://ftp.gnu.org/gnu/
This command lists the contents of the directory at ftp.gnu.org. The output might look like:
drwxr-xr-x 8 1003 1003 4096 Dec 16 2020 0ad
drwxr-sr-x 5 1003 1003 4096 Nov 09 2020 8sync
drwxr-xr-x 2 1003 1003 4096 Jun 05 2015 GNUinfo
drwxr-xr-x 3 1003 1003 4096 Jan 23 2022 GNUnet
...
Creating a Simple Port Scanner
Let's create a simple Bash script to scan a range of ports on a server. Create a new file called port_scanner.sh in your project directory:
nano ~/project/port_scanner.sh
Add the following content to the file:
#!/bin/bash
## Simple port scanner using curl
## Usage: ./port_scanner.sh <hostname> <start_port> <end_port>
hostname=$1
start_port=$2
end_port=$3
echo "Scanning ports $start_port to $end_port on $hostname..."
echo
for port in $(seq $start_port $end_port); do
## Try to connect with a short timeout
curl -s --connect-timeout 1 "$hostname:$port" > /dev/null
## Check if the connection was successful
if [ $? -eq 0 ]; then
echo "Port $port is OPEN"
else
echo "Port $port is closed"
fi
done
echo
echo "Scan complete!"
Save the file by pressing Ctrl+X, then Y, and then Enter.
Make the script executable:
chmod +x ~/project/port_scanner.sh
Now, run the script to scan ports 80-85 on example.com:
~/project/port_scanner.sh example.com 80 85
The output will show which ports are open and which are closed:
Scanning ports 80 to 85 on example.com...
Port 80 is OPEN
Port 81 is closed
Port 82 is closed
Port 83 is closed
Port 84 is closed
Port 85 is closed
Scan complete!
This simple script demonstrates how Curl can be used as a basic port scanning tool. In a real-world scenario, you would want to use specialized tools like nmap for more comprehensive network scanning, but this example shows the versatility of Curl.
Working with HTTPS and Secure Connections
In this step, we'll explore how to use Curl with HTTPS and handle various security-related options.
Understanding HTTPS Connections
HTTPS connections use SSL/TLS protocols to secure data transmission. When you use Curl to connect to an HTTPS site, it verifies the server's SSL certificate by default.
Let's try connecting to a secure website:
curl https://www.google.com
This command connects to Google's HTTPS server and returns the HTML content.
Handling SSL Certificate Verification
Sometimes, you might need to connect to a server with a self-signed or invalid certificate. In such cases, you can use the -k or --insecure option to skip certificate validation:
curl -k https://www.google.com
This command will connect to the site even if the certificate cannot be validated. The output should be similar to the previous command.
Viewing Certificate Information
To examine a website's SSL certificate, use the -v (verbose) option:
curl -v https://www.google.com > /dev/null
This command will display detailed information about the SSL handshake and certificate while sending the actual content to /dev/null. The output includes certificate details:
* Server certificate:
* subject: CN=www.google.com
* start date: ...
* expire date: ...
* subjectAltName: ...
* issuer: CN=GTS CA 1C3; O=Google Trust Services LLC; C=US
* SSL certificate verify ok.
Using Specific TLS Versions
You can specify which TLS version to use with the --tlsv1.X option:
## Force TLS 1.2
curl --tlsv1.2 https://www.google.com > /dev/null
This ensures Curl uses TLS 1.2 for the connection.
Downloading Files Securely
Curl can download files from HTTPS sources. Let's download a file and save it with the -o option:
curl -o ~/project/google_logo.png https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png
This command downloads Google's logo and saves it as google_logo.png in your project directory. Check if the file exists:
ls -l ~/project/google_logo.png
You should see output like:
-rw-r--r-- 1 labex labex 5969 Jun 7 12:34 /home/labex/project/google_logo.png
Specifying Custom Headers for HTTPS
When working with secure APIs, you often need to include authentication headers. Here's how to do that:
curl -H "Authorization: Bearer your_token_here" https://api.example.com
Replace your_token_here with an actual token if you have one. Otherwise, this command will likely return an error or unauthorized message from the server.
Accessing HTTPS on Non-Standard Ports
HTTPS services can run on ports other than 443. To access such services, specify the port in the URL:
curl https://example.com:8443
This command attempts to connect to an HTTPS service on port 8443. Since example.com doesn't have a service on this port, you'll likely see an error:
curl: (7) Failed to connect to example.com port 8443: Connection refused
This demonstrates that Curl can attempt to connect to any port using any protocol, making it a versatile tool for testing network services.
Creating a Comprehensive Port Checking Script
In this final step, we'll create a more sophisticated script that uses Curl to check for common services on a target server.
Understanding Common Service Ports
Different services typically run on standard ports:
- Web (HTTP): Port 80
- Secure Web (HTTPS): Port 443
- FTP: Port 21
- SSH: Port 22
- SMTP (Email): Port 25
- DNS: Port 53
- Database (MySQL): Port 3306
- Database (PostgreSQL): Port 5432
Let's create a script that checks these common ports on a given host.
Creating the Service Checker Script
Create a new file called service_checker.sh in your project directory:
nano ~/project/service_checker.sh
Add the following content to the file:
#!/bin/bash
## Service checker script using curl
## Usage: ./service_checker.sh <hostname>
hostname=$1
if [ -z "$hostname" ]; then
echo "Error: Please provide a hostname."
echo "Usage: ./service_checker.sh <hostname>"
exit 1
fi
echo "Checking common services on $hostname..."
echo
## Function to check a port with appropriate protocol
check_port() {
local port=$1
local service=$2
local protocol=$3
local timeout=2
echo -n "Checking $service (Port $port): "
## Use the appropriate protocol based on the service
if [ "$protocol" = "http" ]; then
curl -s --connect-timeout $timeout "http://$hostname:$port" > /dev/null
elif [ "$protocol" = "https" ]; then
curl -s --connect-timeout $timeout "https://$hostname:$port" > /dev/null
else
## For non-HTTP protocols, just try to connect to the port
curl -s --connect-timeout $timeout "$hostname:$port" > /dev/null
fi
## Check the result
if [ $? -eq 0 ]; then
echo "AVAILABLE"
else
echo "Not available"
fi
}
## Check common services
check_port 80 "Web Server (HTTP)" "http"
check_port 443 "Web Server (HTTPS)" "https"
check_port 21 "FTP Server" "tcp"
check_port 22 "SSH Server" "tcp"
check_port 25 "SMTP Server" "tcp"
check_port 53 "DNS Server" "tcp"
check_port 3306 "MySQL Database" "tcp"
check_port 5432 "PostgreSQL Database" "tcp"
check_port 8080 "Alternative Web Server" "http"
check_port 8443 "Alternative Secure Web Server" "https"
echo
echo "Service check complete!"
Save the file by pressing Ctrl+X, then Y, and then Enter.
Make the script executable:
chmod +x ~/project/service_checker.sh
Running the Service Checker
Now, run the script to check services on a well-known website:
~/project/service_checker.sh example.com
You'll see output similar to this:
Checking common services on example.com...
Checking Web Server (HTTP) (Port 80): AVAILABLE
Checking Web Server (HTTPS) (Port 443): AVAILABLE
Checking FTP Server (Port 21): Not available
Checking SSH Server (Port 22): Not available
Checking SMTP Server (Port 25): Not available
Checking DNS Server (Port 53): Not available
Checking MySQL Database (Port 3306): Not available
Checking PostgreSQL Database (Port 5432): Not available
Checking Alternative Web Server (Port 8080): Not available
Checking Alternative Secure Web Server (Port 8443): Not available
Service check complete!
This output shows that example.com has web servers running on ports 80 and 443, but other common services are not publicly accessible.
Understanding the Results
The results from our script provide valuable information:
Available services: These ports are open and responding to requests, indicating that the corresponding services are running and accessible.
Not available services: These ports might be:
- Closed (no service running)
- Filtered by a firewall
- Running but configured not to respond to generic requests
This information is useful for:
- Network administrators checking service availability
- Security professionals conducting initial reconnaissance
- Developers verifying that their services are properly configured
Modifying the Script
Feel free to modify the script to check additional ports or services. For example, you could add checks for:
- Redis (Port 6379)
- MongoDB (Port 27017)
- RDP (Port 3389)
To add a new service check, simply add another line with the check_port function:
check_port 6379 "Redis Database" "tcp"
This demonstrates the flexibility of Curl as a tool for network service testing and monitoring.
Summary
In this tutorial, you have learned how to use Curl to access different ports on servers and networks. You have:
- Learned the basics of Curl and how to specify ports in URLs
- Used Curl to interact with web servers on standard and non-standard ports
- Explored different HTTP methods and headers
- Worked with HTTPS and secure connections
- Created scripts to check port availability and service status
These skills are valuable for network troubleshooting, system administration, and security testing. Curl's versatility makes it an essential tool in your command-line toolkit.
For additional practice, try:
- Using Curl to interact with REST APIs
- Testing different connection timeouts and retry options
- Exploring more advanced Curl features like cookies, form submissions, and proxy settings
- Enhancing the service checker script with additional protocols and error handling
Remember that while Curl is powerful for testing and basic scanning, specialized tools like Nmap provide more comprehensive network scanning capabilities for professional use.
