How to Rotate Proxies for Web Scraping

Proxy rotation is the single most important technique for scraping at scale. Every IP has a request budget before a target site flags it — rotation spreads your traffic across a large pool so no individual address hits that limit. This guide covers two implementation models, practical code in Python and Node.js, sticky sessions, error handling, and when it makes sense to hand rotation off to a managed service.

Why Rotation Fails Without a Strategy

A naive scraper hits example.com a thousand times from one IP. The site sees a thousand requests in three minutes from the same address — an obvious bot pattern — and returns 429s or silently serves CAPTCHAs. Adding a single proxy doesn't help much if that proxy also exhausts its budget.

Effective rotation means distributing requests across enough IPs that each one stays within a human-looking request rate. The exact threshold varies by site, but a common safe rule is fewer than 30–50 requests per IP per hour for well-defended targets.

Two Models: Proxy List vs. Backconnect Gateway

Proxy list: You hold a collection of host:port pairs (with optional credentials) and pick one programmatically for each request. You control selection logic, can weight by performance, and can mark bad IPs as dead.

Backconnect gateway: The provider gives you a single endpoint (e.g., gate.provider.com:22225). Every TCP connection through it gets a fresh IP from their pool. No selection logic required — just point all your requests at the gateway.

Most residential proxy networks work as backconnect gateways. Datacenter proxies are often sold as lists. Both work; backconnect is simpler to code against and delegates the hard parts (pool management, dead-IP pruning, geolocation routing) to the provider.

Rotating a Proxy List in Python

The pattern below uses requests with a simple round-robin over a list of proxies. In production you'd load the list from a file or API rather than hardcoding it.

import itertools
import requests

RAW_PROXIES = [
    "http://user:pass@proxy1.example.com:8080",
    "http://user:pass@proxy2.example.com:8080",
    "http://user:pass@proxy3.example.com:8080",
]

proxy_pool = itertools.cycle(RAW_PROXIES)

def get(url: str) -> requests.Response:
    proxy = next(proxy_pool)
    proxies = {"http": proxy, "https": proxy}
    return requests.get(url, proxies=proxies, timeout=15)

Round-robin is fine when your list is large and the proxies are roughly equivalent. If some IPs are faster or more reliable, keep a weighted list and track per-proxy success rates to reorder it over time.

Rotating a Backconnect Gateway in Python

With a gateway, every connection already gets a new IP, so you only need one proxy string:

import os
import requests

PROXY_URL = os.environ["PROXY_URL"]   # e.g. http://user:pass@gate.provider.com:22225

def get(url: str) -> requests.Response:
    proxies = {"http": PROXY_URL, "https": PROXY_URL}
    return requests.get(url, proxies=proxies, timeout=20)

Set PROXY_URL as an environment variable so credentials never appear in source code.

Looking for a residential backconnect network? Bright Data's residential proxy network offers automatic rotation across a massive global pool with city-level geo-targeting — point your scraper at their gateway and the pool management is handled for you.

Rotating Proxies in Node.js

Using the built-in https module with a SOCKS/HTTP proxy via https-proxy-agent:

import https from "https";
import { HttpsProxyAgent } from "https-proxy-agent";

const PROXIES = [
  "http://user:pass@proxy1.example.com:8080",
  "http://user:pass@proxy2.example.com:8080",
  "http://user:pass@proxy3.example.com:8080",
];

let idx = 0;
function nextProxy() {
  return PROXIES[idx++ % PROXIES.length];
}

function fetchWithProxy(url) {
  return new Promise((resolve, reject) => {
    const agent = new HttpsProxyAgent(nextProxy());
    https.get(url, { agent }, (res) => {
      let data = "";
      res.on("data", (chunk) => (data += chunk));
      res.on("end", () => resolve({ status: res.statusCode, body: data }));
    }).on("error", reject);
  });
}

// Usage
const result = await fetchWithProxy("https://example.com/");
console.log(result.status);

With a backconnect gateway, replace the PROXIES array with a single process.env.PROXY_URL and remove the rotation logic — every call already gets a fresh IP.

Sticky Sessions: Keeping One IP for a Workflow

Some tasks need the same IP across multiple requests — logging into an account, navigating a multi-step checkout, or filling paginated results that set a session cookie. Most backconnect providers support sticky sessions via a special session parameter appended to the username.

import os, requests

BASE_URL  = os.environ["PROXY_HOST"]  # gate.provider.com:22225
USER      = os.environ["PROXY_USER"]
PASS      = os.environ["PROXY_PASS"]

SESSION_ID = "sess_abc123"  # any unique string per workflow

proxy = f"http://{USER}-session-{SESSION_ID}:{PASS}@{BASE_URL}"
proxies = {"http": proxy, "https": proxy}

# All requests through this proxy share one IP
s = requests.Session()
s.proxies = proxies

s.get("https://example.com/login", data={"user": "me", "pass": "secret"})
s.get("https://example.com/dashboard")  # same IP, session cookie preserved

Generate a new SESSION_ID for each independent workflow and let it expire when the task ends. Check your provider's docs for the exact syntax — Bright Data uses -session-<id> in the username; other providers vary.

Error Handling and Dead-IP Pruning

Proxies fail: timeouts, connection errors, 407 auth failures, and banned IPs that return 403 on the target. A robust scraper should:

1. Catch per-request failures and retry with a different proxy.
2. Track consecutive failures per IP and pull it from the pool after N failures.
3. Log response codes — 403/429 from the target indicates the proxy IP is flagged; a connection error or 407 is a proxy infrastructure problem.

import itertools, requests, time

def get_with_retry(url: str, proxy_pool, retries: int = 3) -> requests.Response | None:
    for _ in range(retries):
        proxy = next(proxy_pool)
        try:
            r = requests.get(
                url,
                proxies={"http": proxy, "https": proxy},
                timeout=15,
            )
            if r.status_code in (403, 429):
                time.sleep(1)
                continue  # IP flagged — try the next one
            return r
        except requests.RequestException:
            time.sleep(1)
    return None  # all retries exhausted

For production scrapers, a simple dead-set prevents a banned IP from ever being retried:

from collections import defaultdict

failure_counts: dict[str, int] = defaultdict(int)
MAX_FAILURES = 5

def get_healthy(url: str, raw_proxies: list[str]) -> requests.Response | None:
    healthy = [p for p in raw_proxies if failure_counts[p] < MAX_FAILURES]
    for proxy in healthy:
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)
            if r.status_code < 400:
                return r
            failure_counts[proxy] += 1
        except requests.RequestException:
            failure_counts[proxy] += 1
    return None

Choosing the Right Proxy Tier for Rotation

Not every target needs expensive residential IPs. See Residential vs. Datacenter vs. Mobile Proxies for the full breakdown, but the quick rule:

• Sites with no anti-bot protection: Rotate datacenter proxies — fast, cheap, works well.
• E-commerce, travel, news: Residential rotation. Datacenter ranges are often pre-blocked.
• Social media, ticketing, sneakers: Residential or mobile. These platforms fingerprint heavily; see How to Avoid Getting Blocked for the complete toolkit.
• Amazon, Walmart, eBay: Residential is the safe default; see our guides on Amazon product tracking and eBay scraping for target-specific tips.

For provider comparisons, the Bright Data residential proxies review and proxy provider comparisons cover the main options side by side.

When to Let the Provider Handle Rotation

Managing your own proxy list requires buying IPs in bulk, monitoring health, refreshing expired IPs, and tuning rotation intervals per target. For large-scale or time-sensitive scraping, delegating all of this to a managed rotating endpoint often costs less than the engineering time to do it yourself.

Bright Data's residential network is a backconnect gateway backed by a large global pool. You set the geo-target and session length in the proxy URL string; the provider handles pool health, dead-IP retirement, and city-level routing. For targets that also need JavaScript rendering and CAPTCHA solving, the Bright Data Scraping Browser adds those layers on top of the same proxy infrastructure.

FAQ

What is proxy rotation and why does it matter?

Proxy rotation means automatically switching the IP address your scraper uses between requests. Without rotation, a single IP sends hundreds of requests in a short window and gets rate-limited or banned. By cycling through a pool of IPs, each address stays under the detection threshold.

How often should I rotate to a new proxy?

For high-volume scraping, rotate on every request so no single IP accumulates a detectable footprint. For tasks that need a consistent identity — staying logged in, filling a multi-step form — use a sticky session to keep the same IP for the duration of that workflow, then rotate when the session ends.

Do I need residential proxies to rotate effectively?

It depends on the target. Datacenter IPs are cheap and fast, and rotating through a large pool works fine on sites with basic rate-limiting. For sites protected by Cloudflare, DataDome, or similar systems, residential or ISP proxies rotate with far fewer blocks because each IP looks like a genuine home user.

What is the difference between a proxy list and a backconnect gateway?

A proxy list is a set of individual IP:port pairs you manage yourself — you pick which one to use per request. A backconnect (rotating) gateway is a single endpoint that automatically assigns a fresh IP from the provider's pool on each connection, so you write no rotation logic yourself.