Proxy & Profile Best Practices

How to configure proxies, browser profiles and the CAPTCHA solver so your Scraping Browser runs stay fast, consistent and unblocked. This page is about choices, not mechanics - it assumes you can already create a session and connect to it. For endpoints, parameters and code, see the Scraping Browser API reference.

1. The proxy is your identity, not just a pipe

A proxy is not a transport detail. It is the channel through which every site, API and LLM perceives you. Before a single line of your script runs, the target has already decided what to show you based on the IP alone:

What the IP determines	Effect on your results
Country / region / city / ISP	Which catalogue, currency, language and prices you see. Google Shopping, marketplaces and LLM answers are all geo-shaped, and results can differ between regions of the same country.
Network type (residential / mobile)	Whether you are treated as a person or as automation. Datacenter IPs are not supported here - and would not survive these targets anyway.
IP reputation and history	Whether you get content, a CAPTCHA, a soft block, or degraded/decoy results.

The most common cause of “bad data” is not the scraper. It is a mismatched or unstable IP producing a different page than the one you expected. Rule: decide the geo you actually want before writing selectors. Target the location of the audience whose results you are trying to reproduce. Verify it before you trust the data. On the first run of any new configuration, check where you are actually coming out - open an IP-echo page, or watch the session’s live view - and confirm the country, region and currency the target is serving you. Most “the data looks wrong” reports are a session that was never in the location it was configured for.

2. Choosing the proxy type

The Scraping Browser runs on residential and mobile proxies only. Datacenter and static/ISP proxies are not supported - they are the first thing a serious anti-bot system filters out, so the browser is built around real-network IPs by design. Residential (30M+ IPs) - real IPs on real consumer ISPs. The default for search engines, marketplaces, PLA/Shopping data and anything with serious bot protection. Mobile (295K+ IPs) - IPs from mobile carrier networks (4G/5G). The highest trust level available, because carrier-grade NAT means thousands of real users share the same address and blocking it is expensive for the target. The pool is smaller, so reserve it for where it counts: the hardest targets, mobile-first platforms and social apps, or when residential is still getting challenged. Both pools are filtered for low fraud score before an IP ever reaches you, so what you get is a real, clean address that looks like a real user to the target. That filtering is the product. What no provider can remove is the nature of the network itself: these are real devices, and a real device can go offline mid-session. It happens occasionally - the exception, not the rule - so build for it rather than assuming a permanently open connection. The Sticky session type absorbs it for you by substituting an equivalent IP (§3). Rule of thumb: start on residential. Move to mobile when the target keeps challenging you, or when it is a mobile-first platform.

3. Proxy settings

Everything in this section lives under Proxy Settings in the dashboard.

Location settings

Target the location explicitly - Country, then Region and City when the data is local (prices, availability, local packs). ISP and ZIP targeting narrow it further when you need to reproduce a very specific user’s view. Do not leave geography to the pool if it affects the output. Country and region show a quality badge (High / Medium / Low) indicating pool depth for that location. Prefer High where you can - a thin pool means fewer IPs to rotate through and faster reuse of the same addresses.

Session type

Mode (dashboard label)	What it does	Use it when
Rotating	A new IP on every request / every new session.	High-volume, stateless collection where each request stands alone and you want to spread across many addresses.
Sticky - Keep IP as long as possible (Super sticky sessions)	Holds one IP for the session. If that IP goes offline, the system automatically substitutes another IP matching the same targeting (country, region, city) so the session keeps running.	The default for most jobs. You want one stable identity, but you care more about the run finishing than about it being literally the same address end to end.
No rotating	Pinned to one specific IP. If it drops, nothing is substituted - the session waits for that IP to come back rather than silently switching you to another one.	Logged-in accounts, long multi-step flows, and anything where an IP change mid-session would break trust or invalidate a solved challenge. Gives you full manual control: you decide when to switch.

The trade-off is continuity vs. identity. Sticky keeps the job alive and absorbs the fact that residential and mobile IPs drop; No rotating keeps the identity perfectly intact but hands you the drop to deal with yourself. Rotating per request looks like a hundred different people arriving at the same funnel in the same second. Sticky looks like one person browsing - and your cookies, consent state and any solved CAPTCHA stay valid for the rest of the run. Rule: one job = one location = one sticky session. Rotate between jobs, not inside them. Use No rotating whenever the session carries a logged-in state or a solved challenge.

Protocol

Two options: HTTP and SOCKS5. Use SOCKS5 for protected sites. It is the more reliable choice against targets with serious bot protection - Google in particular - and we have measured noticeably more consistent results with it. HTTP is fine for undefended endpoints and simple fetches. Rule: protected target → SOCKS5. Make it your default and only drop to HTTP if something specifically requires it.

IP filter mode

Controls how the pool picks an IP for you.

Mode	Optimises for	Use it when
Quality (default)	Lowest fraud score. Filters the pool down to clean IPs with good reputation.	Leave it here. This is what gets you through anti-bot checks - clean history beats raw speed on any protected target.
Quality + Speed	Clean IPs, then the fastest among them.	You need volume but still care about getting through. A reasonable compromise for large jobs on moderately protected sites.
Speed	Lowest latency.	Undefended endpoints where throughput is the only thing that matters.
Max pool size	The widest possible set of IPs, with the least filtering.	You need maximum IP diversity - very large rotating jobs - and the target does not scrutinise IP reputation.

Rule: stay on Quality (default) unless you have measured a reason not to. Loosening the filter trades fraud score for speed or pool size, and fraud score is exactly what protected sites check.

IP version

Mixed IPv4 and IPv6 by default. Switch to IPv4 only if the target handles IPv6 badly or serves different content over it - some sites still treat IPv6 ranges as higher risk.

4. Browser profiles: solve the CAPTCHA once

This is the single biggest speed and success-rate lever in the product. Under Browser Profile Settings, the Browser state toggle decides everything here:

One-time - state is discarded when the session ends. Every run starts as a stranger and pays for the CAPTCHA again.
Persistent - state is kept and reused. This is the setting you want for any repeated job.

A persistent browser profile keeps the real Chrome state between sessions - cookies, local storage, history, consent flags and the trust signals a site has already granted you. Consequences:

The first run on a hard target may hit a challenge. You solve it once.
The next run reuses those cookies, so the site recognises a returning visitor and, in most cases, does not challenge you again.
Runs get dramatically faster, because you skip the solve entirely instead of paying 1-3 minutes for it every time.
Behaviour becomes reproducible: same profile, same IP, same page.

Pin the profile to the proxy. A profile carrying cookies issued to a US residential IP that suddenly appears from Germany is a contradiction, and it is exactly the pattern anti-bot systems look for. Bind each profile to one location and keep it on the same (or closest available) IP every time. Rules:

Set Browser state → Persistent. One-time profiles throw away everything this section is about.
Create one profile per target + geo combination, and reuse it. Do not create a fresh profile per run.
Store the profile ID alongside the proxy config you used with it, and always pair them again.
Never move a warmed profile to a different country.
Warm a profile with a couple of light runs before pointing it at your heaviest job.

Other browser settings worth knowing

Setting	Recommendation
Extra Stealth - managed fingerprint and anti-detection	On. Together with the CAPTCHA Solver it does the heavy lifting on bot checks.
CAPTCHA Solver	On. See §6 for how long to wait for it.
Browser mode	Headful (recommended). Headless is a detection signal on protected sites.
Ad Blocker / Popup Blocker	On by default. Blocks ads and unnecessary resources, which speeds up page loads. Turn the Ad Blocker off if a target breaks or you actually need the ad/PLA units to render.
Cloudflare Web Bot Auth	Off by default. Signs your requests for participating Cloudflare sites - turn it on only when you are working with a site that expects an identified bot.
Idle timeout	5 minutes, fixed. A session with no activity for that long closes - so keep long waits (like a CAPTCHA solve) filled with actual polling (§6).

5. Arrive the way a real user would

Never deep-link straight to the page you actually want. Jumping from a search engine - or from nothing at all - directly into a product page, a listing or a detail URL is one of the clearest automation signals a platform has, and it is a common reason for an instant block or challenge on an otherwise perfect setup. A real customer does not materialise on amazon.com/dp/B0XXXXXXX. They land on the homepage, search, filter, and click through. Reproduce that path:

Open the homepage first. Let the site set its cookies, consent state, session and regional context.
Search from the site’s own search box - type the query you actually want, don’t rewrite the URL.
Apply filters and sorting in the UI rather than constructing a query-string that no human would produce.
Click through to the product from the results list.
Pace it. Leave roughly 1-3 seconds between actions and let each page settle before the next click. That is enough to stop looking like a script without burning your session budget - a five-step path costs you well under a minute.

The referrer chain, the cookies picked up along the way and the timing between steps all end up in the platform’s risk score. Deep-linking discards every one of them. This compounds with §4: the navigation warms the profile, the profile keeps that warmth for the next run, and after a few clean passes the site stops treating you as a first-time stranger. Rule: homepage → search → filter → product. Direct-to-product URLs only from a profile that has already earned trust on that domain.

6. The CAPTCHA solver: it works, but give it time

The built-in solver handles challenges reliably - including Google’s /sorry/ interstitial. The thing to internalise is it is not instant.

Typical clear: a few seconds.
Hard challenges: ~1 to 3 minutes, occasionally longer.
It generally does get through - the failures we see in the wild are almost always client-side timeouts firing while the solver was still working.

So the default 30-60 second timeout most scripts ship with is the wrong number here. Rule: set the CAPTCHA wait to at least 240 seconds (3-4 minutes). Poll the page for the challenge clearing rather than sleeping a fixed amount, and only then continue. Poll actively while you wait - don’t sleep. This is the trap: the session closes after 5 minutes of inactivity, and that limit is fixed (§7). A blind sleep(240) is 4 minutes of nothing happening, and the session can die right as the solver finishes. Poll the page every few seconds instead - that keeps the session alive and lets you continue the moment the challenge clears. The 5-minute idle window is therefore your real ceiling: treat ~4 minutes as the practical maximum for a single CAPTCHA wait. And budget the session as a whole - a hard solve can eat 3 of your 30 available minutes. Combine this with §4: with a warmed profile you will usually not reach the challenge at all, which is why the profile is worth setting up.

7. Default limits

These limits are fixed - they are not settings you can change. Plan your jobs around them:

Limit	Default
Browser profiles	10
Concurrent active sessions	50
Maximum session duration	30 minutes
Idle timeout	5 minutes
Session recording retention	3 days

What this means in practice:

Budget the 30-minute duration. A hard CAPTCHA can eat 1-3 minutes of it (§6). Size each job to fit inside one session, and continue longer work in a new session on the same persistent profile so it does not start cold.
10 profiles is a design constraint, not a nuisance. It maps naturally to one profile per target + geo (§4). Reuse them; don’t burn them on one-off runs.
50 concurrent sessions is your real parallelism ceiling - size your worker pool to it and queue the rest.
Watch the 5-minute idle timeout. A session with nothing happening on it closes. Never park a session between steps, and never block on a bare sleep() - poll instead (§6).
Pull recordings within 3 days. If you are debugging a failed run, download the recording before it ages out.

Need more than this? Write to support with your use case. Limit increases are reviewed case by case rather than granted on request, so explain what you are running and why the default does not fit.

8. Recipe: Google Search and Shopping (PLA) scraping

Google is the hardest common target, so it makes a good worked example. This configuration reliably pulls clean product data - titles, prices and merchant URLs - from both the inline shopping unit and the Shopping tab, with Extra Stealth and the CAPTCHA Solver doing the work.

Configuration checklist

#	Setting	Why
1	Do not use the in-dashboard Playground. Run your own script against the CDP API.	The Playground sandbox caps execution at 60 seconds. Google’s CAPTCHA can take longer than that to clear, so a Playground run is killed mid-solve and never finishes. Connecting your own local or server-side script directly to the session’s CDP URL has no such limit.
2	CAPTCHA wait ≥ 240s	See §6. Google challenges have taken ~1-3 minutes in our runs.
3	Protocol: `SOCKS5` (not HTTP)	Noticeably more consistent results against Google specifically. Set it under Proxy Settings when creating the session (§3).
4	Extra Stealth: ON and CAPTCHA Solver: ON	These two do the heavy lifting on Google’s bot checks.
5	Residential (or mobile) + Sticky + IP filter mode: Quality + the exact country/region/city you want prices for	Google Shopping output is geo-dependent, and a clean low-fraud-score IP is what gets you past the bot check; see §1 and §3.
6	Reuse a warmed browser profile	After the first solve, most subsequent runs skip the CAPTCHA wall entirely.
7	Poll while waiting for the CAPTCHA, never `sleep()`	The session closes after 5 idle minutes, so a blind sleep can kill it mid-solve (§6).
8	Turn the Ad Blocker OFF	PLA units are ads. The Ad Blocker is on by default and can strip the very Shopping results you came for - if your extraction returns zero products on a page that clearly has them, check this first.

One practical note on the flow: if the regular search returns no inline shopping unit, retry the query on the Shopping tab (&tbm=shop). Expect the /sorry/ wall on either step - handle it the same way both times.

9. Quick reference

proxy:
  type: residential        # residential or mobile - the only supported types
  protocol: socks5         # http | socks5 - socks5 for protected sites
  ip_filter_mode: quality  # quality (default) | quality+speed | speed | max-pool-size
  ip_version: mixed        # mixed ipv4+ipv6 | ipv4-only
  country: us              # always pin explicitly
  region: <optional>       # state/province - results differ within a country
  city: <optional>         # pin when prices/availability are local
  isp: <optional>          # narrow further if you need a specific provider's view
  session_type: sticky     # rotating | sticky | no-rotating
                           # no-rotating for logged-in / long flows

browser:
  profile: <reuse-the-same-id>   # one profile per target+geo, bound to the proxy above
  browser_state: persistent      # NOT one-time - this is what keeps the cookies
  browser_mode: headful
  extra_stealth: true
  captcha_solver: true

navigation:
  path: homepage -> search -> filter -> product   # never deep-link to a detail URL
  pacing: let each page settle before the next click

limits:                          # fixed - support reviews increases case by case
  browser_profiles: 10
  concurrent_sessions: 50
  session_duration_minutes: 30
  idle_timeout_minutes: 5
  recording_retention_days: 3

timeouts:
  captcha_wait_seconds: 240      # never 30-60
  execution: run-your-own-script # Playground caps at 60s

10. Troubleshooting

Symptom	Likely cause	Fix
Wrong prices, currency or catalogue	IP is in the wrong geo	Pin country, region and city explicitly (§1, §3)
Results change between identical runs	Rotating IP or a new profile each run	Sticky session + reuse one profile (§3, §4)
IP silently changed mid-session	Smart rotation replaced a dropped IP	Switch to sticky with no replacement if the session must keep one address (§3)
Session stalls after an IP drops	No-replacement stickiness waiting for the IP	Expected behaviour - wait, or switch proxy manually (§3)
Script dies “on the CAPTCHA”	Client timeout fired mid-solve	Raise the wait to ≥240s (§6)
Run always killed around 60s	Running in the Playground	Connect your own script to the CDP URL (§8.1)
Google blocks persistently	HTTP protocol, or stealth/solver off	Switch to SOCKS5, enable both (§8.3-8.4)
Blocked despite a clean setup	IP filter mode loosened past Quality	Return to Quality (default) for a low fraud score (§3)
Profile forgets cookies between runs	Browser state left on One-time	Switch to Persistent (§4)
Session closed while nothing was happening	5-minute idle timeout (fixed)	Poll instead of sleeping; keep the session busy (§6)
Zero products extracted from a page that has them	Ad Blocker stripped the PLA units	Turn the Ad Blocker off for Shopping scraping (§8)
Job doesn’t fit in one session	30-minute maximum	Continue in a new session on the same persistent profile (§7)
Connection drops mid-run	Normal residential/mobile IP behaviour	Retry within the same sticky session; keep the profile
Every run hits a CAPTCHA	Fresh profile each time	Warm one profile and reuse it (§4)
Instant block on a product/detail page	Deep-linked straight to the URL	Go homepage → search → filter → click through (§5)
Session ends around the 30-minute mark	Maximum session duration reached	Split the job across sessions; carry state in the profile (§7)
Cannot create another profile	10-profile limit reached	Reuse existing profiles; a raise is a support request, reviewed case by case (§7)
New sessions queue or fail to start	Concurrent-session limit in use	Cap your worker pool at the limit and queue the rest (§7)
Recording no longer available	Older than 3 days	Download recordings while debugging, before retention expires (§7)

Platform API

Product APIs

Proxy & Profile Best Practices

1. The proxy is your identity, not just a pipe

2. Choosing the proxy type

3. Proxy settings

Location settings

Session type

Protocol

IP filter mode

IP version

4. Browser profiles: solve the CAPTCHA once

Other browser settings worth knowing

5. Arrive the way a real user would

6. The CAPTCHA solver: it works, but give it time

7. Default limits

8. Recipe: Google Search and Shopping (PLA) scraping

Configuration checklist

9. Quick reference

10. Troubleshooting

​1. The proxy is your identity, not just a pipe

​2. Choosing the proxy type

​3. Proxy settings

​Location settings

​Session type

​Protocol

​IP filter mode

​IP version

​4. Browser profiles: solve the CAPTCHA once

​Other browser settings worth knowing

​5. Arrive the way a real user would

​6. The CAPTCHA solver: it works, but give it time

​7. Default limits

​8. Recipe: Google Search and Shopping (PLA) scraping

​Configuration checklist

​9. Quick reference

​10. Troubleshooting

1. The proxy is your identity, not just a pipe

2. Choosing the proxy type

3. Proxy settings

Location settings

Session type

Protocol

IP filter mode

IP version

4. Browser profiles: solve the CAPTCHA once

Other browser settings worth knowing

5. Arrive the way a real user would

6. The CAPTCHA solver: it works, but give it time

7. Default limits

8. Recipe: Google Search and Shopping (PLA) scraping

Configuration checklist

9. Quick reference

10. Troubleshooting