Proxy pool sizing
How Many Proxies Do I Need for Scraping?
A practical way to size a scraping proxy pool by request rate, sessions, target sensitivity and reserve.
Practical estimate
For scraping, start by estimating safe concurrent requests per proxy, then multiply by target count, region count and retry reserve. A small stable private pool is usually better than a large untested list.
Workload
Pool
Reserve
BuyProxies guide
A good proxy count is not the biggest count. It is the smallest pool that keeps sessions stable, avoids repeated failures and leaves enough reserve for retries.
Simple formula
Starting pool = active sessions ÷ safe sessions per proxy × location multiplier × 1.25 reserve.
For example, if you need 40 active workers and you want no more than 2 active workers per proxy, start with 20 proxies, then add reserve for retries and bad routes.
Example pool sizes
| Workload | Starting point | Why |
|---|---|---|
| Small scraper | 5-10 proxies | Enough to test routes, request rate and timeout handling. |
| Medium product/price monitoring | 25-75 proxies | Depends on target count, markets and freshness requirements. |
| Large crawler | 100+ proxies | Segment by target and region; do not treat one big pool as one workload. |
When to add more proxies
Add more proxies only when the current pool is healthy but capacity is the bottleneck. If many requests fail because credentials are wrong, the country is wrong or the target is blocking the workflow, a bigger pool just spreads the same mistake over more IPs. First fix connection quality, request pacing and session rules, then increase capacity.
A useful rule is to scale in visible steps. Increase the pool, run the same workload, compare error rate and keep the change only if useful results increase without a matching increase in blocks, timeouts or support work. For location-sensitive workflows, treat every country as its own pool instead of averaging all proxies together.
How to size the pool
- Measure requests per minute and active workers.
- Decide safe sessions or workers per proxy.
- Split by country or target domain if needed.
- Add 25-30% reserve for retries and bad routes.
- Review error rate before increasing concurrency.
A simple calculation example
Suppose the workload needs 30 active sessions and you are comfortable with 2 active sessions per proxy. The base pool is 15 proxies. If the job runs in two countries, split that into two location pools instead of one mixed pool. Then add reserve capacity for retries, maintenance and unexpected target friction. With a 25% reserve, that 15-proxy base becomes about 19 proxies before you round up for location coverage.
This calculation is not meant to be perfect. It gives you a safe starting number that can be measured. After the first run, compare successful results, retries, timeouts and manual fixes. If success rate is high but the queue is too slow, add proxies. If error rate is high, fix the workflow before buying more capacity.
Sizing mistakes
- Buying a large pool before testing target behavior.
- Rotating too aggressively when sessions need stability.
- Ignoring timeout and 403 patterns.
- Using one global pool for very different targets.
FAQ
Can I scrape with one proxy?
For tiny tests, yes. For sustained scraping, one proxy becomes a bottleneck and a single point of failure.
Should I rotate scraping proxies?
Rotate only when the target workflow benefits from it. Some workflows need sticky sessions.
How much reserve should I keep?
Start with at least 25% reserve, then adjust after measuring real failures.