How to set custom headers in a Python requests call

Introduction

In the world of web development and data retrieval, understanding how to set custom headers in Python requests is a crucial skill. This tutorial will guide you through the process of adding custom headers to your Python requests, unlocking a wide range of possibilities for web scraping, API interactions, and more.

Custom headers allow you to control how your requests are processed by web servers and APIs. They can help you authenticate requests, specify content types, manage caching, and even mimic browser behavior. By the end of this tutorial, you will have the practical knowledge to customize your HTTP requests to meet specific requirements.

Understanding HTTP Headers and the Requests Library

Before we start writing code, let's understand what HTTP headers are and why they are important.

What are HTTP Headers?

HTTP headers are key-value pairs sent in requests and responses between clients and servers. They provide additional information about the request or response, such as the content type, authentication credentials, or client information.

For example, when you visit a website, your browser automatically includes headers like User-Agent (identifying your browser), Accept (specifying which content types you can handle), and more.

Installing the Requests Library

The Python Requests library makes working with HTTP requests simple. Let's make sure it's installed in our environment:

1. Open a terminal in the WebIDE and run:

pip install requests

You should see output indicating requests is being installed or is already installed:

Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (2.28.1)
...

Creating Your First Python Script with Requests

Let's create a simple Python script to test the requests library:

1. Click on the Explorer icon in the left sidebar of the WebIDE
2. Right-click in the file explorer and select "New File"
3. Name the file test_requests.py
4. Add the following code to the file:

import requests

## Make a simple GET request to a test website
response = requests.get('https://httpbin.org/get')

## Print the response status code
print(f"Status Code: {response.status_code}")

## Print the response headers (what the server sent back)
print("\nResponse Headers:")
for header, value in response.headers.items():
    print(f"{header}: {value}")

## Print the response content
print("\nResponse Content:")
print(response.text[:300] + "...")  ## Showing just the first 300 characters

5. Run the script by opening a terminal and executing:

python test_requests.py

You should see output similar to:

Status Code: 200

Response Headers:
Date: Wed, 12 Apr 2023 10:15:23 GMT
Content-Type: application/json
Content-Length: 267
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Response Content:
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.28.1",
    "X-Amzn-Trace-Id": "Root=1-643..."
  },
  "origin": "123.45.67.89",
  "url": "https://httpbin.org/get"
}
...

This output shows:

1. The server responded with status code 200 (Success)
2. The headers the server sent back
3. The actual content of the response (in JSON format)

Notice that the server records the headers it received from us, including the automatically generated User-Agent header that identifies our request as coming from the Python requests library.

In the next step, we'll learn how to customize these headers to have more control over our requests.

Setting Basic Custom Headers in Python Requests

Now that we understand what HTTP headers are and have tested the requests library, let's learn how to set custom headers in our requests.

Why Set Custom Headers?

There are several reasons to set custom headers:

1. Authentication: Many APIs require authentication tokens sent in headers
2. User-Agent: Change how you identify yourself to websites
3. Content-Type: Specify the format of data you're sending
4. Accept: Indicate which content types you can handle
5. Custom Headers: Some APIs require specific custom headers

Setting Headers in Requests

Let's create a new Python script to practice setting custom headers:

1. In the WebIDE, create a new file named custom_headers.py
2. Add the following code:

import requests

## Define the URL we'll make the request to
url = 'https://httpbin.org/headers'

## Define custom headers in a dictionary
custom_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'X-Custom-Header': 'Hello World',
    'Accept-Language': 'en-US,en;q=0.9'
}

## Make the request with our custom headers
response = requests.get(url, headers=custom_headers)

## Print the status code
print(f"Status Code: {response.status_code}")

## Print the response content (which shows the headers the server received)
print("\nHeaders received by server:")
print(response.json())

3. Run the script:

python custom_headers.py

You should see output similar to:

Status Code: 200

Headers received by server:
{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.9', 'Host': 'httpbin.org', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36', 'X-Custom-Header': 'Hello World', 'X-Amzn-Trace-Id': 'Root=1-6430a123-abc123def456'}}

What's Happening?

In this script, we:

1. Created a dictionary called custom_headers containing three headers:

   • User-Agent: We changed it from the default Python requests value to mimic a web browser
   • X-Custom-Header: A completely custom header we invented
   • Accept-Language: Specifies preferred languages for content

2. Passed our headers to requests.get() using the headers parameter

3. The response from httpbin.org/headers shows us exactly what headers were received, confirming our custom headers were sent successfully

Modifying Headers for a Specific Use Case

Let's create another example where we pretend to be a mobile device. This can be useful when testing mobile-responsive websites:

1. Create a new file named mobile_headers.py
2. Add the following code:

import requests

url = 'https://httpbin.org/headers'

## Header to simulate an iPhone
mobile_headers = {
    'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
}

print("Making request with mobile device headers...")
response = requests.get(url, headers=mobile_headers)

## Print the response content
print(response.json())

3. Run the script:

python mobile_headers.py

You should see output similar to:

Making request with mobile device headers...
{'headers': {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1', 'X-Amzn-Trace-Id': 'Root=1-6430a456-abc123def456'}}

This shows that we're successfully telling the server we're an iPhone by setting a specific User-Agent header.

Working with Authentication Headers

One of the most common uses of custom headers is for authentication with web services and APIs. Many APIs require you to include authentication information in your request headers.

Understanding Authentication Headers

There are several common authentication methods:

1. API Keys: Simple tokens included in headers or URL parameters
2. Basic Authentication: Base64-encoded username and password
3. Bearer Tokens: Used in OAuth and JWT authentication
4. Custom Authentication: Proprietary authentication schemes

Let's explore how to implement these authentication methods with Python requests.

API Key Authentication

Many APIs allow access using API keys. These are typically included in headers or as URL parameters:

1. Create a new file named api_key_auth.py
2. Add the following code:

import requests

url = 'https://httpbin.org/headers'

## Simulating an API key authentication
api_key = 'my_fake_api_key_12345'

## Add the API key to the headers
headers = {
    'X-API-Key': api_key,
    'User-Agent': 'My API Client/1.0'
}

## Make the request
response = requests.get(url, headers=headers)

## Print results
print(f"Status Code: {response.status_code}")
print("\nHeaders received by server:")
print(response.json())

3. Run the script:

python api_key_auth.py

You should see output similar to:

Status Code: 200

Headers received by server:
{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'My API Client/1.0', 'X-API-Key': 'my_fake_api_key_12345', 'X-Amzn-Trace-Id': 'Root=1-6430a789-abc123def456'}}

This confirms we've successfully sent an API key in the X-API-Key header.

Basic Authentication

Basic authentication involves sending a username and password encoded in base64 format. The Requests library makes this easy:

1. Create a new file named basic_auth.py
2. Add the following code:

import requests

## URL that requires basic authentication
url = 'https://httpbin.org/basic-auth/user/pass'

## Method 1: Using the auth parameter (recommended)
print("Method 1: Using the auth parameter")
response = requests.get(url, auth=('user', 'pass'))
print(f"Status Code: {response.status_code}")
print(response.json())

## Method 2: Setting the Authorization header manually
print("\nMethod 2: Setting the Authorization header manually")
import base64

## Create the basic auth string: "username:password" encoded in base64
auth_string = base64.b64encode('user:pass'.encode('utf-8')).decode('utf-8')
headers = {
    'Authorization': f'Basic {auth_string}'
}

response = requests.get(url, headers=headers)
print(f"Status Code: {response.status_code}")
print(response.json())

3. Run the script:

python basic_auth.py

You should see output similar to:

Method 1: Using the auth parameter
Status Code: 200
{'authenticated': True, 'user': 'user'}

Method 2: Setting the Authorization header manually
Status Code: 200
{'authenticated': True, 'user': 'user'}

Both methods successfully authenticated with the service:

• Method 1 uses the built-in auth parameter, which is more convenient
• Method 2 shows how basic authentication works "under the hood" by manually creating the Authorization header

Bearer Token Authentication

Bearer token authentication is commonly used in OAuth 2.0 and JWT-based systems:

1. Create a new file named bearer_auth.py
2. Add the following code:

import requests

url = 'https://httpbin.org/headers'

## Simulate a bearer token (in a real app this would come from an OAuth process)
bearer_token = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIn0.Q6CM1qIQkXA_DqyJq9MI0-mnRRCRR3c0SIM-Nul5MZs'

## Add the Authorization header with the Bearer token
headers = {
    'Authorization': f'Bearer {bearer_token}'
}

## Make the request
response = requests.get(url, headers=headers)

## Print results
print(f"Status Code: {response.status_code}")
print("\nHeaders received by server:")
print(response.json())

3. Run the script:

python bearer_auth.py

You should see output similar to:

Status Code: 200

Headers received by server:
{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIn0.Q6CM1qIQkXA_DqyJq9MI0-mnRRCRR3c0SIM-Nul5MZs', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6430a901-abc123def456'}}

This shows a successful request with a Bearer token included in the Authorization header.

Bearer tokens are typically obtained after a successful authentication process with an OAuth 2.0 server, but in this example, we're using a sample token for demonstration purposes.

Practical Applications and Best Practices

Now that you have learned how to set various types of custom headers, let's explore some practical applications and best practices for working with headers in real-world scenarios.

Sending and Receiving JSON Data

When working with modern APIs, JSON is the most common data format. Let's see how to properly set headers for JSON requests:

1. Create a new file named json_requests.py
2. Add the following code:

import requests
import json

url = 'https://httpbin.org/post'  ## This endpoint accepts POST requests

## Data to send
data = {
    'name': 'John Doe',
    'email': 'john@example.com',
    'message': 'Hello, world!'
}

## Set appropriate headers for JSON
headers = {
    'Content-Type': 'application/json',
    'Accept': 'application/json'
}

## Method 1: Using the json parameter (recommended)
print("Method 1: Using the json parameter")
response = requests.post(url, json=data, headers=headers)
print(f"Status Code: {response.status_code}")
print(response.json()['headers'])  ## Show the headers received
print(f"\nData sent (as received by server): {response.json()['json']}")

## Method 2: Manually converting to JSON
print("\nMethod 2: Manually converting to JSON")
json_data = json.dumps(data)
response = requests.post(url, data=json_data, headers=headers)
print(f"Status Code: {response.status_code}")
print(response.json()['headers'])  ## Show the headers received
print(f"\nData sent (as received by server): {response.json()['json']}")

3. Run the script:

python json_requests.py

You should see output similar to:

Method 1: Using the json parameter
Status Code: 200
{'Accept': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '72', 'Content-Type': 'application/json', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6430ab12-abc123def456'}

Data sent (as received by server): {'name': 'John Doe', 'email': 'john@example.com', 'message': 'Hello, world!'}

Method 2: Manually converting to JSON
Status Code: 200
{'Accept': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '72', 'Content-Type': 'application/json', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6430ab13-abc123def456'}

Data sent (as received by server): {'name': 'John Doe', 'email': 'john@example.com', 'message': 'Hello, world!'}

Note that both methods work, but Method 1 is more convenient because the Requests library handles the JSON conversion for you.

Creating a Reusable Session with Default Headers

If you need to make multiple requests with the same headers, using a Session object can save time and make your code cleaner:

1. Create a new file named session_headers.py
2. Add the following code:

import requests

## Create a session object
session = requests.Session()

## Set default headers for all requests made with this session
session.headers.update({
    'User-Agent': 'MyCustomApp/1.0',
    'Accept-Language': 'en-US,en;q=0.9',
    'X-API-Key': 'my_session_api_key'
})

## Make a request using the session - it will include our default headers
print("First request with session:")
response = session.get('https://httpbin.org/headers')
print(response.json())

## Add a custom header just for this request
print("\nSecond request with additional header:")
response = session.get('https://httpbin.org/headers',
                      headers={'X-Custom-Header': 'Just for this request'})
print(response.json())

## Original headers remain unchanged for future requests
print("\nThird request (original headers only):")
response = session.get('https://httpbin.org/headers')
print(response.json())

3. Run the script:

python session_headers.py

You should see output similar to:

First request with session:
{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.9', 'Host': 'httpbin.org', 'User-Agent': 'MyCustomApp/1.0', 'X-API-Key': 'my_session_api_key', 'X-Amzn-Trace-Id': 'Root=1-6430ab45-abc123def456'}}

Second request with additional header:
{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.9', 'Host': 'httpbin.org', 'User-Agent': 'MyCustomApp/1.0', 'X-API-Key': 'my_session_api_key', 'X-Custom-Header': 'Just for this request', 'X-Amzn-Trace-Id': 'Root=1-6430ab46-abc123def456'}}

Third request (original headers only):
{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.9', 'Host': 'httpbin.org', 'User-Agent': 'MyCustomApp/1.0', 'X-API-Key': 'my_session_api_key', 'X-Amzn-Trace-Id': 'Root=1-6430ab47-abc123def456'}}

Using sessions is a best practice for several reasons:

1. Improved performance (connection pooling)
2. Consistent headers across requests
3. Automatic cookie handling
4. Ability to add per-request customization when needed

Best Practices for Working with Headers

To wrap up, here are some best practices to follow when working with HTTP headers:

1. Use Session Objects for multiple requests to the same domain
2. Set Content-Type Headers correctly for the data you're sending
3. Handle Authentication Properly - use built-in auth when possible
4. Keep Sensitive Information Secure - don't hardcode API keys or tokens
5. Follow API Documentation carefully for required headers

Here's a final example that demonstrates these best practices:

1. Create a new file named best_practices.py
2. Add the following code:

import requests
import os

## In a real application, get these from environment variables or a secure configuration
## For this example, we're keeping it simple
API_KEY = os.environ.get('API_KEY', 'default_api_key')
BASE_URL = 'https://httpbin.org'

def create_api_client():
    """Create a reusable session with default headers."""
    session = requests.Session()

    ## Set common headers
    session.headers.update({
        'User-Agent': 'MyApp/1.0',
        'X-API-Key': API_KEY,
        'Accept': 'application/json'
    })

    return session

def get_data(client, endpoint):
    """Make a GET request to the specified endpoint."""
    url = f"{BASE_URL}/{endpoint}"
    response = client.get(url)
    return response.json()

def post_data(client, endpoint, data):
    """Make a POST request with JSON data."""
    url = f"{BASE_URL}/{endpoint}"
    response = client.post(url, json=data)
    return response.json()

## Usage example
def main():
    ## Create the API client
    client = create_api_client()

    ## GET request example
    print("Making GET request...")
    get_response = get_data(client, 'headers')
    print(f"Headers sent: {get_response['headers']}")

    ## POST request example
    print("\nMaking POST request...")
    data = {'name': 'John', 'age': 30}
    post_response = post_data(client, 'post', data)
    print(f"Data received by server: {post_response['json']}")

if __name__ == '__main__':
    main()

3. Run the script:

python best_practices.py

You should see output similar to:

Making GET request...
Headers sent: {'Accept': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'MyApp/1.0', 'X-API-Key': 'default_api_key', 'X-Amzn-Trace-Id': 'Root=1-6430ab78-abc123def456'}

Making POST request...
Data received by server: {'name': 'John', 'age': 30}

This example demonstrates a structured approach to making HTTP requests with custom headers, following best practices like creating reusable sessions, organizing code into functions, and avoiding hardcoded sensitive information.

Summary

In this tutorial, you have learned how to work with custom headers in Python requests, which is an essential skill for effective web communication and API integration.

You now understand:

• What HTTP headers are and why they are important for web requests
• How to set custom headers for individual requests
• Various authentication methods using headers (API keys, Basic auth, Bearer tokens)
• How to properly work with JSON data and content type headers
• Best practices for managing headers with session objects

These skills will enable you to interact with any web API more effectively, access protected resources, control how your requests are processed, and build more sophisticated web applications.

As you continue to develop your Python skills, remember that understanding HTTP basics like headers, status codes, and content types will help you troubleshoot issues and implement more advanced features in your web applications.