How to Use Proxies for Web Scraping

Why Websites Block Scrapers

When you send a request from a single IP address at high frequency, the server easily identifies that traffic as non-human. Rate limiting, IP blacklisting, CAPTCHAs, and even legal threats are common defences. Without a proxy strategy, your scraper will hit a wall after a few hundred requests.

The core problem is IP-based rate limiting: each request carries your public IP, and if too many come too quickly, the server assumes abuse. Proxies solve this by distributing requests across many IPs, making the traffic appear organic.

How Proxies Help Avoid IP Bans

A proxy sits between your scraper and the target server. The server sees the proxy's IP, not yours. By rotating through a pool of proxies, you stay under the per-IP rate limit. Even with aggressive scraping, no single IP experiences high volume.

In essence, proxies act as a buffer that absorbs the blame. The server sees many different visitors from different locations, each making a modest number of requests. That's much harder to flag as a bot.

Residential vs Datacenter Proxies

Not all proxies are equal. The choice between residential and datacenter proxies depends on the target site's sensitivity and your budget.

Residential proxies come from real ISPs and appear as genuine home users. They are harder to detect and block, but more expensive and slower. Use them for large e‑commerce sites or social media platforms that aggressively block datacenter IP ranges.
Datacenter proxies are hosted on cloud servers. They are cheap, fast, and abundant, but many sites have their IP ranges flagged. They work well for less protected sites or when you need high throughput.

ProxyVerity's proxies by type list helps you find both kinds of verified proxies.

Rotating vs Sticky Proxies

Rotation determines how often your IP changes. Two common modes:

Rotating – a new IP for every request (or every few requests). Best for scraping many pages from the same site, as it spreads requests thinly.
Sticky – the same IP stays for a session (e.g., 10 minutes). Useful when you need to maintain a session or logged‑in state, like scraping a private dashboard after login.

Most proxy providers let you toggle rotation. For beginners, rotating is safer: request A comes from IP 1, request B from IP 2, etc. The server never sees sustained traffic from one source.

Setting Request Headers and User‑Agents

Proxies solve the IP problem, but servers also inspect request headers. A default Python requests library sends a tell‑tale python-requests/2.x User‑Agent. That alone can get you blocked.

Always set realistic User‑Agent strings (e.g., from a recent browser) and mimic typical headers like Accept, Accept-Language, and Referer. Rotate User‑Agents along with IPs.

“If your request doesn't look like it came from a real browser, you're already at a disadvantage.”

Use a rotating list of User‑Agents, one per request. Many libraries and services offer this; even a simple list of ten strings works wonders.

Adding Delays and Randomisation

Even with proxies and headers, sending requests too fast triggers behavioural detection. The server sees that every 0.5 seconds exactly a request arrives. Human visitors don't behave like that.

Add random delays between requests. For example, after each page, wait between 2 and 6 seconds, randomised. This smooths the traffic pattern and keeps you under rate limits. For larger scrapes, consider crawling politely: one request per second per IP is safe for most sites.

Ethics, Legality, and `robots.txt`

Using proxies does not absolve you from ethical scraping. Always check the site's robots.txt and terms of service. If the disallows scraping, respect that — ignoring it can lead to legal action. Public proxies are third‑party infrastructure, often free and unencrypted; never send sensitive data through them.

Our free proxy list is tested for status and speed, but you must still use them responsibly. Scrape at reasonable rates, cache where possible, and give back by not overwhelming servers.

A Practical Example

Here's a short Python snippet using requests with a rotating proxy and random delay. It fetches a sample product page with a common User‑Agent.

import requestsimport randomimport timeproxies = [    { 'http': 'http://proxy1:port', 'https': 'http://proxy1:port' },    { 'http': 'http://proxy2:port', 'https': 'http://proxy2:port' },    # ... add more from your proxy list]user_agents = [    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...',    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ...',    # ... more UAs]url = 'https://example.com/products'for i in range(10):    proxy = random.choice(proxies)    headers = { 'User-Agent': random.choice(user_agents) }    try:        response = requests.get(url, proxies=proxy, headers=headers, timeout=10)        print(f'{i+1}: {response.status_code} from {proxy["http"]}')    except Exception as e:        print(f'{i+1}: Error - {e}')    time.sleep(random.uniform(2, 6))

Replace the proxy strings with IPs from your provider or from our verified proxies by location list. For better anonymity, use a proxy checker to confirm that your proxies are working and not leaking DNS or your real IP.

Final Thoughts

Proxies are one piece of a larger puzzle: combine IP rotation with headers, delays, and ethical practices. Start small, test thoroughly, and scale slowly. With the right approach, you can scrape at scale without triggering bans — and without harming the sites you rely on.