ProxyVerity
Tutorial

How to Use Proxies in Python Requests: Configuration and IP Rotation

Learn how to configure proxies in Python Requests, including authentication, SOCKS5, environment variables, and advanced IP rotation with retry logic for resilient web scraping.

RB
By Roman Bulatov
Proxy & Network Infrastructure Expert · · 5 min read

Why Proxy Configuration Matters

When scraping the web or accessing geo-restricted content, your IP address often determines success or failure. Proxies let you route requests through an intermediate server, masking your origin and distributing load. Python's requests library makes proxy integration straightforward, but real-world use demands more than a single static proxy. This article walks you from basic setup to a resilient, rotating proxy system.

Basic Proxy Configuration with the proxies Dict

The simplest way to use a proxy is passing a dict to the proxies parameter. Keys are protocols (http, https) and values are proxy URLs.

import requests

proxies = {
    'http': 'http://proxy.example.com:8080',
    'https': 'http://proxy.example.com:8080',
}
response = requests.get('http://httpbin.org/ip', proxies=proxies)
print(response.json())

If you only need a proxy for one protocol, omit the other. Requests will fall back to the http proxy for https if https is not specified but that often fails with HTTPS errors—always specify both.

Authenticated Proxies

Many paid or private proxies require a username and password. Include them directly in the URL:

proxies = {
    'http': 'http://user:pass@proxy.example.com:8080',
    'https': 'http://user:pass@proxy.example.com:8080',
}

Requests handles the Proxy-Authorization header automatically. For more complex authentication (e.g., NTLM), consider the requests-ntlm library.

SOCKS5 vs SOCKS5h

SOCKS proxies are often faster for TCP traffic, but require the PySocks package (pip install PySocks). Two variants exist: socks5 resolves DNS locally (on the client), while socks5h resolves it remotely (on the proxy).

# Remote DNS (recommended for privacy)
proxies = {
    'http': 'socks5h://proxy.example.com:1080',
    'https': 'socks5h://proxy.example.com:1080',
}

Use socks5h to avoid leaking DNS queries to your ISP or local network. If the proxy supplies DNS, it may also cache or filter results—be aware of that trade-off.

Environment Variables

Requests also respects standard environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. This is handy for globally applying a proxy without modifying code.

import os
os.environ['HTTP_PROXY'] = 'http://proxy.example.com:8080'
os.environ['HTTPS_PROXY'] = 'http://proxy.example.com:8080'
os.environ['NO_PROXY'] = 'localhost,127.0.0.1'

Note that the proxies dict takes precedence over environment variables when both are set.

Using Sessions for Persistent Proxy Settings

If you make multiple requests, a Session object keeps the proxy configuration, connection pools, and cookies. Set session.proxies once and reuse the session.

session = requests.Session()
session.proxies = proxies
response = session.get('http://httpbin.org/ip')

This avoids re-authenticating on every request and improves performance.

Advanced IP Rotation: Rotating Proxy Pools

Static proxies get blocked quickly. Rotating through a pool spreads requests across IPs. Use itertools.cycle to loop endlessly.

import itertools

proxy_pool = itertools.cycle([
    'http://proxy1.com:8080',
    'http://proxy2.com:8080',
    'http://proxy3.com:8080',
])

for _ in range(10):
    proxy = next(proxy_pool)
    response = requests.get('http://httpbin.org/ip', proxies={'http': proxy})

For larger pools, read proxies from a file or query a dynamic source like ProxyVerity's free proxy list. Always validate proxies before heavy use—a quick test with a known endpoint avoids silent failures.

Retry Logic with Exponential Backoff

Transient failures (timeouts, 5xx errors) are inevitable. Requests can retry automatically using an HTTPAdapter with a Retry policy.

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
session.mount('http://', HTTPAdapter(max_retries=retries))
session.mount('https://', HTTPAdapter(max_retries=retries))

The backoff_factor causes waits of 0.5s, 1s, 2s, 4s, 8s after successive failures—spreading requests helps avoid triggering rate limits.

Handling Timeouts and Errors

Always set a timeout to prevent hanging. Catch proxy-specific exceptions to handle corrupt or unresponsive proxies gracefully.

try:
    response = requests.get('http://example.com', proxies=proxies, timeout=10)
    response.raise_for_status()
except requests.exceptions.Timeout:
    print('Request timed out')
except requests.exceptions.ProxyError:
    print('Proxy error – remove this proxy')
except requests.exceptions.RequestException as e:
    print(e)

Use timeout as a tuple to separate connect and read timeouts: timeout=(3, 8).

Building a Resilient Scraper

Combine IP rotation, retries, and error handling into a reusable fetcher.

import itertools
import time
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

proxy_list = ['http://proxy1.com:8080', 'http://proxy2.com:8080']
proxy_cycle = itertools.cycle(proxy_list)

session = requests.Session()
retries = Retry(total=3, backoff_factor=0.5, status_forcelist=[500, 502, 503])
session.mount('http://', HTTPAdapter(max_retries=retries))

def fetch(url):
    for _ in range(len(proxy_list)):
        proxy = next(proxy_cycle)
        try:
            response = session.get(url, proxies={'http': proxy}, timeout=5)
            return response
        except (requests.exceptions.ProxyError, requests.exceptions.Timeout):
            continue
    return None

This function tries each proxy once, then gives up. For more robustness, log failures, remove dead proxies from the pool, and integrate with ProxyVerity's checker to refresh the list.

Choosing the Right Proxies

Not all proxies are equal. Free public proxies often have short lifetimes, low speed, or expose your data. For production use, consider reputable providers or build your own pool from verified free proxies filtered by location and anonymity level. Our IP detect tool helps confirm the proxy is working correctly.

Safety and Legality

Proxies pass your traffic through a third party. Public proxies can log, modify, or steal data—never send sensitive information through them. Always use HTTPS end-to-end to encrypt the payload, though the proxy still sees the destination IP and handshake. Legally, scraping must respect robots.txt and the site's terms of service. Rotating IPs to bypass rate limits may violate these terms. Use proxies responsibly and for ethical purposes only.

Back to ProxyHub