Introduction
Welcome to the Interacting with Web Apps lab. In the world of cybersecurity and web automation, interacting with web applications programmatically is a fundamental skill. Whether you are scraping data, testing for vulnerabilities, or automating repetitive tasks, you need a reliable way to send and receive HTTP requests.
In this lab, you will learn how to use the powerful Python requests library. You will write scripts to send HTTP GET and POST requests, extract important information from server responses, automate form submissions, and manage session cookies to maintain authentication state.
Compared with raw sockets, requests works at a higher abstraction level. It handles many HTTP details for you, such as headers, encoding, redirects, and cookies. That makes it ideal for security automation where you care about web behavior and workflows more than packet-by-packet mechanics.
Send HTTP GET Requests
In this step, you will learn how to send a basic HTTP GET request using Python. The GET method is used to request data from a specified resource, much like how your web browser requests a webpage when you type in a URL.
Think of this as the simplest form of web automation: ask for a resource and inspect the response. Many reconnaissance, testing, and monitoring tasks start exactly here.
First, ensure you are in your project directory. This is where you will create all your Python scripts for this lab.
cd ~/project
Now, let's create a Python script to send a GET request to a local web server that has been prepared for you. Use the nano text editor to create a file named get_request.py:
nano get_request.py
Type the following Python code into the editor:
import requests
url = 'http://127.0.0.1:5000/get'
response = requests.get(url)
print("Response Body:")
print(response.text)
Save the file and exit the editor (in nano, press Ctrl+O, Enter to save, and Ctrl+X to exit).
The code above imports the requests library, defines the target URL, and uses the requests.get() function to send the request. The server's response is stored in the response variable, and we print the raw text of the response body.
Run your script using the Python interpreter:
python3 get_request.py
You should see an output similar to this:
Response Body:
{"message":"GET request successful"}
Extract HTTP Status and Headers
In this step, you will learn how to extract more detailed information from an HTTP response, specifically the status code and the headers.
The response body is only part of the story. Status codes tell you how the server understood the request, and headers describe metadata such as content type, caching policy, cookies, or server software. Good automation looks at all three layers: code, headers, and body.
When a server responds to an HTTP request, it includes a status code (like 200 for OK, or 404 for Not Found) and headers that provide metadata about the response (like the content type or server software).
Create a new file named check_status.py in your project directory:
nano check_status.py
Add the following code to the file:
import requests
url = 'http://127.0.0.1:5000/get'
response = requests.get(url)
print(f"Status Code: {response.status_code}")
if response.status_code == 200:
print("The request was successful!")
print("\nResponse Headers:")
print(f"Content-Type: {response.headers.get('Content-Type')}")
print(f"Server: {response.headers.get('Server')}")
Save the file and exit the editor.
In this script, response.status_code retrieves the integer status code returned by the server. The response.headers is a dictionary-like object containing all the HTTP headers sent back by the server. We use the .get() method to safely extract specific headers.
Run the script to see the extracted information:
python3 check_status.py
Your output should look like this:
Status Code: 200
The request was successful!
Response Headers:
Content-Type: application/json
Server: Werkzeug/3.0.1 Python/3.10.12
Automate Form Submissions (POST)
In this step, you will learn how to send an HTTP POST request. Unlike GET requests, which are used to retrieve data, POST requests are used to send data to the server. This is commonly used for submitting web forms, such as login credentials.
This distinction reflects HTTP semantics. In practice, GET is usually used for reading resources, while POST is often used for actions that submit or change state. Understanding that difference helps you predict how applications behave and how your scripts should interact with them.
We will write a script to simulate submitting a login form with a username and password.
Create a new file named post_request.py:
nano post_request.py
Enter the following code:
import requests
url = 'http://127.0.0.1:5000/post'
## Define the form data as a dictionary
form_data = {
'username': 'admin',
'password': 'secret'
}
## Send the POST request with the data parameter
response = requests.post(url, data=form_data)
print(f"Status Code: {response.status_code}")
print(f"Response Body: {response.text}")
Save the file and exit the editor.
Here, we define a dictionary called form_data containing our credentials. We then use requests.post() and pass our dictionary to the data parameter. The requests library automatically formats this data as a standard web form submission.
Run the script to submit the form:
python3 post_request.py
You should see a successful login message:
Status Code: 200
Response Body: {"message":"Login successful"}
You can try changing the password in the script to an incorrect value (like wrongpass) and running it again to see how the server responds with a 401 Unauthorized status code.
Manage Session Cookies
In this final step, you will learn how to manage session cookies. HTTP is a stateless protocol, meaning each request is independent. To remember a user after they log in, web applications use cookies.
If you use requests.get() or requests.post() directly, cookies are not saved between requests. To maintain a session (like staying logged in), you must use a Session object provided by the requests library.
Create a file named session_request.py:
nano session_request.py
Add the following code:
import requests
## Create a Session object
session = requests.Session()
login_url = 'http://127.0.0.1:5000/login'
profile_url = 'http://127.0.0.1:5000/profile'
print("Attempting to access profile BEFORE login...")
response1 = session.get(profile_url)
print(f"Status: {response1.status_code}, Body: {response1.text.strip()}")
print("\nLogging in to establish a session...")
## The session object will automatically store cookies returned by the server
session.post(login_url)
print("\nAttempting to access profile AFTER login...")
## The session object will automatically send the stored cookies with this request
response2 = session.get(profile_url)
print(f"Status: {response2.status_code}, Body: {response2.text.strip()}")
Save the file and exit the editor.
In this script, requests.Session() creates a persistent session. When we send the POST request to /login, the server sets a session cookie. The Session object stores this cookie and automatically includes it in the subsequent GET request to /profile, allowing us to access the protected route.
Run the script:
python3 session_request.py
Observe the output. You will see that the first attempt fails because we are not logged in, but the second attempt succeeds because the session cookie was managed automatically:
Attempting to access profile BEFORE login...
Status: 401, Body: {"message":"Unauthorized"}
Logging in to establish a session...
Attempting to access profile AFTER login...
Status: 200, Body: {"message":"Welcome to your profile!"}
Summary
You have learned how to use the Python requests library to automate web interactions. You started by sending basic HTTP GET requests and extracting important metadata like status codes and headers. You then advanced to sending HTTP POST requests to automate form submissions. Finally, you learned how to use Session objects to manage cookies and maintain authentication state across multiple requests.
These skills form the foundation for many advanced tasks in web scraping, API integration, and automated security testing.
