What is the best way to send a GET request with Python request headers?

General Discussions
1

What is the best way to send a GET request with Python request headers?

I used Fiddler to capture a GET request, and I want to resend the exact request with Python. Below is the captured request:

GET https://example.com/api/content/v1/products/search?page=20&page_size=25&q=&type=image HTTP/1.1
Host: example.com
Connection: keep-alive
Search-Version: v3
Accept: application/json
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
Referer: https://example.com/search/?q=&type=image&page=20
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

How can I send this GET request with the exact Python request headers using Python?

2

In my experience, the simplest and most efficient way to send a GET request with python request headers is by using the requests library. It’s pretty intuitive and makes the whole process straightforward. You just pass the headers as a dictionary to the headers parameter in the requests.get() function. Here’s how you can do it:

import requests

url = "https://example.com/api/content/v1/products/search"
params = {
    "page": 20,
    "page_size": 25,
    "q": "",
    "type": "image"
}

headers = {
    "Host": "example.com",
    "Connection": "keep-alive",
    "Search-Version": "v3",
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
    "Referer": "https://example.com/search/?q=&type=image&page=20",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9"
}

response = requests.get(url, headers=headers, params=params)

print(response.text)

This matches the request exactly, including the headers and query parameters, so it’s a perfect solution for most needs when dealing with python request headers.

3

Oh, I totally agree, @charity-majors. But if you’re like me and prefer more granular control, maybe because you’re dealing with more complex requests or prefer using built-in libraries, then you might want to check out the http.client module. It gives you a lower-level approach, where you can manage the connection and headers more explicitly.

Here’s how it works:

import http.client

conn = http.client.HTTPSConnection("example.com")

headers = {
    "Host": "example.com",
    "Connection": "keep-alive",
    "Search-Version": "v3",
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
    "Referer": "https://example.com/search/?q=&type=image&page=20",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9"
}

url = "/api/content/v1/products/search?page=20&page_size=25&q=&type=image"

conn.request("GET", url, headers=headers)

response = conn.getresponse()
print(response.read().decode())

conn.close()

In this case, you control the connection and raw request much more directly, which can be useful when you need that extra flexibility with python request headers.

4

Both methods are solid, but if you’re in a situation where you need to stick to Python’s standard library and prefer something a bit more manual, you can also use urllib.request. While it’s not as user-friendly as requests, it’s handy when you’re working with libraries that don’t support requests. Plus, if you’re looking to implement your own logic around how requests are sent, it offers that flexibility.

Here’s the implementation:

import urllib.request

url = "https://example.com/api/content/v1/products/search?page=20&page_size=25&q=&type=image"

headers = {
    "Host": "example.com",
    "Connection": "keep-alive",
    "Search-Version": "v3",
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
    "Referer": "https://example.com/search/?q=&type=image&page=20",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9"
}

request = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(request)

print(response.read().decode())

This method works well when you need to rely solely on Python’s built-in modules and still want to customize how you handle the python request headers. It gives you more flexibility if you don’t mind a little extra setup.