Skip to main content

Proxy & Profile Best Practices

How to configure proxies, browser profiles and the CAPTCHA solver in the Scraping Browser so your runs stay fast, consistent and unblocked.

Written by NodeMaven

1. The proxy is your identity, not just a pipe

A proxy is not a transport detail. It is the channel through which every site, API and LLM perceives you. Before a single line of your script runs, the target has already decided what to show you based on the IP alone:

What the IP determines

Effect on your results

Country / region / city / ISP

Which catalogue, currency, language and prices you see. Google Shopping, marketplaces and LLM answers are all geo-shaped, and results can differ between regions of the same country.

Network type (residential / mobile)

Whether you are treated as a person or as automation. Datacenter IPs are not supported here — and would not survive these targets anyway.

IP reputation & history

Whether you get content, a CAPTCHA, a soft block, or degraded/decoy results.

The most common cause of "bad data" is not the scraper. It is a mismatched or unstable IP producing a different page than the one you expected.

Rule: decide the geo you actually want before writing selectors. Target the location of the audience whose results you are trying to reproduce.

Verify it before you trust the data. On the first run of any new configuration, check where you are actually coming out — open an IP-echo page, or watch the session's live view — and confirm the country, region and currency the target is serving you. Most "the data looks wrong" reports are a session that was never in the location it was configured for.


2. Choosing the proxy type

The Scraping Browser runs on residential and mobile proxies only. Datacenter and static/ISP proxies are not supported — they are the first thing a serious anti-bot system filters out, so the browser is built around real-network IPs by design.

Residential (30M+ IPs) - real IPs on real consumer ISPs. The default for search engines, marketplaces, PLA/Shopping data and anything with serious bot protection.

Mobile (295K+ IPs) - IPs from mobile carrier networks (4G/5G). The highest trust level available, because carrier-grade NAT means thousands of real users share the same address and blocking it is expensive for the target. The pool is smaller, so reserve it for where it counts: the hardest targets, mobile-first platforms and social apps, or when residential is still getting challenged.

Both pools are filtered for low fraud score before an IP ever reaches you, so what you get is a real, clean address that looks like a real user to the target. That filtering is the product.

What no provider can remove is the nature of the network itself: these are real devices, and a real device can go offline mid-session. It happens occasionally — the exception, not the rule — so build for it rather than assuming a permanently open connection. The Sticky session type absorbs it for you by substituting an equivalent IP (§3).

Rule of thumb: start on residential. Move to mobile when the target keeps challenging you, or when it is a mobile-first platform.


3. Proxy settings

Everything in this section lives under Proxy Settings in the dashboard.

Location settings

Target the location explicitly — Country, then Region and City when the data is local (prices, availability, local packs). ISP and ZIP targeting narrow it further when you need to reproduce a very specific user's view. Do not leave geography to the pool if it affects the output.

Country and region show a quality badge (High / Medium / Low) indicating pool depth for that location. Prefer High where you can - a thin pool means fewer IPs to rotate through and faster reuse of the same addresses.

Session type

Mode (dashboard label)

What it does

Use it when

Rotating

A new IP on every request / every new session.

High-volume, stateless collection where each request stands alone and you want to spread across many addresses.

StickyKeep IP as long as possible (Super sticky sessions)

Holds one IP for the session. If that IP goes offline, the system automatically substitutes another IP matching the same targeting (country, region, city) so the session keeps running.

The default for most jobs. You want one stable identity, but you care more about the run finishing than about it being literally the same address end to end.

No rotating

Pinned to one specific IP. If it drops, nothing is substituted — the session waits for that IP to come back rather than silently switching you to another one.

Logged-in accounts, long multi-step flows, and anything where an IP change mid-session would break trust or invalidate a solved challenge. Gives you full manual control: you decide when to switch.

The trade-off is continuity vs. identity. Sticky keeps the job alive and absorbs the fact that residential and mobile IPs drop; No rotating keeps the identity perfectly intact but hands you the drop to deal with yourself.

Rotating per request looks like a hundred different people arriving at the same funnel in the same second. Sticky looks like one person browsing - and your cookies, consent state and any solved CAPTCHA stay valid for the rest of the run.

Rule: one job = one location = one sticky session. Rotate between jobs, not inside them. Use No rotating whenever the session carries a logged-in state or a solved challenge.

Protocol

Two options: HTTP and SOCKS5.

Use SOCKS5 for protected sites. It is the more reliable choice against targets with serious bot protection — Google in particular — and we have measured noticeably more consistent results with it. HTTP is fine for undefended endpoints and simple fetches.

Rule: protected target → SOCKS5. Make it your default and only drop to HTTP if something specifically requires it.

IP filter mode

Controls how the pool picks an IP for you.

Mode

Optimises for

Use it when

Quality (default)

Lowest fraud score. Filters the pool down to clean IPs with good reputation.

Leave it here. This is what gets you through anti-bot checks - clean history beats raw speed on any protected target.

Quality + Speed

Clean IPs, then the fastest among them.

You need volume but still care about getting through. A reasonable compromise for large jobs on moderately protected sites.

Speed

Lowest latency.

Undefended endpoints where throughput is the only thing that matters.

Max pool size

The widest possible set of IPs, with the least filtering.

You need maximum IP diversity - very large rotating jobs - and the target does not scrutinise IP reputation.

Rule: stay on Quality (default) unless you have measured a reason not to. Loosening the filter trades fraud score for speed or pool size, and fraud score is exactly what protected sites check.

IP version

Mixed IPv4 and IPv6 by default. Switch to IPv4 only if the target handles IPv6 badly or serves different content over it — some sites still treat IPv6 ranges as higher risk.


4. Browser profiles: solve the CAPTCHA once

This is the single biggest speed and success-rate lever in the product.

Under Browser Profile Settings, the Browser state toggle decides everything here:

  • One-time — state is discarded when the session ends. Every run starts as a stranger and pays for the CAPTCHA again.

  • Persistent — state is kept and reused. This is the setting you want for any repeated job.

A persistent browser profile keeps the real Chrome state between sessions — cookies, local storage, history, consent flags and the trust signals a site has already granted you. Consequences:

  • The first run on a hard target may hit a challenge. You solve it once.

  • The next run reuses those cookies, so the site recognises a returning visitor and, in most cases, does not challenge you again.

  • Runs get dramatically faster, because you skip the solve entirely instead of paying 1–3 minutes for it every time.

  • Behaviour becomes reproducible: same profile, same IP, same page.

Pin the profile to the proxy. A profile carrying cookies issued to a US residential IP that suddenly appears from Germany is a contradiction, and it is exactly the pattern anti-bot systems look for. Bind each profile to one location and keep it on the same (or closest available) IP every time.

Rules:

  • Set Browser state → Persistent. One-time profiles throw away everything this section is about.

  • Create one profile per target + geo combination, and reuse it. Do not create a fresh profile per run.

  • Store the profile ID alongside the proxy config you used with it, and always pair them again.

  • Never move a warmed profile to a different country. - Warm a profile with a couple of light runs before pointing it at your heaviest job.

Other browser settings worth knowing

Setting

Recommendation

Extra Stealth — managed fingerprint and anti-detection

On. Together with the CAPTCHA Solver it does the heavy lifting on bot checks.

CAPTCHA Solver

On. See §6 for how long to wait for it.

Browser mode

Headful (recommended). Headless is a detection signal on protected sites.

Ad Blocker / Popup Blocker

On by default. Blocks ads and unnecessary resources, which speeds up page loads. Turn the Ad Blocker off if a target breaks or you actually need the ad/PLA units to render.

Cloudflare Web Bot Auth

Off by default. Signs your requests for participating Cloudflare sites - turn it on only when you are working with a site that expects an identified bot.

Idle timeout

5 minutes, fixed. A session with no activity for that long closes - so keep long waits (like a CAPTCHA solve) filled with actual polling (§6).


5. Arrive the way a real user would

Never deep-link straight to the page you actually want. Jumping from a search engine - or from nothing at all - directly into a product page, a listing or a detail URL is one of the clearest automation signals a platform has, and it is a common reason for an instant block or challenge on an otherwise perfect setup.

A real customer does not materialise on amazon.com/dp/B0XXXXXXX. They land on the homepage, search, filter, and click through. Reproduce that path:

  1. Open the homepage first. Let the site set its cookies, consent state, session and regional context.

  2. Search from the site's own search box — type the query you actually want, don't rewrite the URL.

  3. Apply filters and sorting in the UI rather than constructing a query-string that no human would produce.

  4. Click through to the product from the results list.

  5. Pace it. Leave roughly 1–3 seconds between actions and let each page settle before the next click. That is enough to stop looking like a script without burning your session budget — a five-step path costs you well under a minute.

The referrer chain, the cookies picked up along the way and the timing between steps all end up in the platform's risk score. Deep-linking discards every one of them.

This compounds with §4: the navigation warms the profile, the profile keeps that warmth for the next run, and after a few clean passes the site stops treating you as a first-time stranger.

Rule: homepage → search → filter → product. Direct-to-product URLs only from a profile that has already earned trust on that domain.


6. The CAPTCHA solver: it works, but give it time

The built-in solver handles challenges reliably — including Google's /sorry/ interstitial. The thing to internalise is it is not instant.

  • Typical clear: a few seconds.

  • Hard challenges: ~1 to 3 minutes, occasionally longer.

  • It generally does get through — the failures we see in the wild are almost always client-side timeouts firing while the solver was still working.

So the default 30–60 second timeout most scripts ship with is the wrong number here.

Rule: set the CAPTCHA wait to at least 240 seconds (3–4 minutes). Poll the page for the challenge clearing rather than sleeping a fixed amount, and only then continue.

Poll actively while you wait - don't sleep. This is the trap: the session closes after 5 minutes of inactivity, and that limit is fixed (§7). A blind sleep(240) is 4 minutes of nothing happening, and the session can die right as the solver finishes. Poll the page every few seconds instead - that keeps the session alive and lets you continue the moment the challenge clears.

The 5-minute idle window is therefore your real ceiling: treat ~4 minutes as the practical maximum for a single CAPTCHA wait. And budget the session as a whole - a hard solve can eat 3 of your 30 available minutes.

Combine this with §4: with a warmed profile you will usually not reach the challenge at all, which is why the profile is worth setting up.


7. Default limits

These limits are fixed - they are not settings you can change. Plan your jobs around them:

Limit

Default

Browser profiles

10

Concurrent active sessions

50

Maximum session duration

30 minutes

Idle timeout

5 minutes

Session recording retention

3 days

What this means in practice:

  • Budget the 30-minute duration. A hard CAPTCHA can eat 1–3 minutes of it (§6). Size each job to fit inside one session, and continue longer work in a new session on the same persistent profile so it does not start cold.

  • 10 profiles is a design constraint, not a nuisance. It maps naturally to one profile per target + geo (§4). Reuse them; don't burn them on one-off runs.

  • 50 concurrent sessions is your real parallelism ceiling — size your worker pool to it and queue the rest.

  • Watch the 5-minute idle timeout. A session with nothing happening on it closes. Never park a session between steps, and never block on a bare sleep() — poll instead (§6).

  • Pull recordings within 3 days. If you are debugging a failed run, download the recording before it ages out.

Need more than this? Write to support with your use case. Limit increases are reviewed case by case rather than granted on request, so explain what you are running and why the default does not fit.


8. Recipe: Google Search & Shopping (PLA) scraping

Google is the hardest common target, so it makes a good worked example. This configuration reliably pulls clean product data — titles, prices and merchant URLs — from both the inline shopping unit and the Shopping tab, with Extra Stealth and the CAPTCHA Solver doing the work.

Configuration checklist

#

Setting

Why

1

Do not use the in-dashboard Playground. Run your own script against the CDP API.

The Playground sandbox caps execution at 60 seconds. Google's CAPTCHA can take longer than that to clear, so a Playground run is killed mid-solve and never finishes. Connecting your own local or server-side script directly to the session's CDP URL has no such limit.

2

CAPTCHA wait ≥ 240s

See §6. Google challenges have taken ~1–3 minutes in our runs.

3

Protocol: SOCKS5 (not HTTP)

Noticeably more consistent results against Google specifically. Set it under Proxy Settings when creating the session (§3).

4

Extra Stealth: ON and CAPTCHA Solver: ON

These two do the heavy lifting on Google's bot checks.

5

Residential (or mobile) + Sticky + IP filter mode: Quality + the exact country/region/city you want prices for

Google Shopping output is geo-dependent, and a clean low-fraud-score IP is what gets you past the bot check; see §1 and §3.

6

Reuse a warmed browser profile

After the first solve, most subsequent runs skip the CAPTCHA wall entirely.

7

Poll while waiting for the CAPTCHA, never sleep()

The session closes after 5 idle minutes, so a blind sleep can kill it mid-solve (§6).

8

Turn the Ad Blocker OFF

PLA units are ads. The Ad Blocker is on by default and can strip the very Shopping results you came for — if your extraction returns zero products on a page that clearly has them, check this first.

One practical note on the flow: if the regular search returns no inline shopping unit, retry the query on the Shopping tab (&tbm=shop). Expect the /sorry/ wall on either step — handle it the same way both times.


9. Quick reference

proxy:
type: residential # residential or mobile - the only supported types
protocol: socks5 # http | socks5 - socks5 for protected sites
ip_filter_mode: quality # quality (default) | quality+speed | speed | max-pool-size
ip_version: mixed # mixed ipv4+ipv6 | ipv4-only
country: us # always pin explicitly
region: <optional> # state/province - results differ within a country
city: <optional> # pin when prices/availability are local
isp: <optional> # narrow further if you need a specific provider's view
session_type: sticky # rotating | sticky | no-rotating
# no-rotating for logged-in / long flows

browser:
profile: <reuse-the-same-id> # one profile per target+geo, bound to the proxy above
browser_state: persistent # NOT one-time - this is what keeps the cookies
browser_mode: headful
extra_stealth: true
captcha_solver: true

navigation:
path: homepage -> search -> filter -> product # never deep-link to a detail URL
pacing: let each page settle before the next click

limits: # fixed - support reviews increases case by case
browser_profiles: 10
concurrent_sessions: 50
session_duration_minutes: 30
idle_timeout_minutes: 5
recording_retention_days: 3

timeouts:
captcha_wait_seconds: 240 # never 30-60
execution: run-your-own-script # Playground caps at 60s

10. Troubleshooting

Symptom

Likely cause

Fix

Wrong prices, currency or catalogue

IP is in the wrong geo

Pin country, region and city explicitly (§1, §3)

Results change between identical runs

Rotating IP or a new profile each run

Sticky session + reuse one profile (§3, §4)

IP silently changed mid-session

Smart rotation replaced a dropped IP

Switch to sticky with no replacement if the session must keep one address (§3)

Session stalls after an IP drops

No-replacement stickiness waiting for the IP

Expected behaviour — wait, or switch proxy manually (§3)

Script dies "on the CAPTCHA"

Client timeout fired mid-solve

Raise the wait to ≥240s (§6)

Run always killed around 60s

Running in the Playground

Connect your own script to the CDP URL (§8.1)

Google blocks persistently

HTTP protocol, or stealth/solver off

Switch to SOCKS5, enable both (§8.3–8.4)

Blocked despite a clean setup

IP filter mode loosened past Quality

Return to Quality (default) for a low fraud score (§3)

Profile forgets cookies between runs

Browser state left on One-time

Switch to Persistent (§4)

Session closed while nothing was happening

5-minute idle timeout (fixed)

Poll instead of sleeping; keep the session busy (§6)

Zero products extracted from a page that has them

Ad Blocker stripped the PLA units

Turn the Ad Blocker off for Shopping scraping (§8)

Job doesn't fit in one session

30-minute maximum

Continue in a new session on the same persistent profile (§7)

Connection drops mid-run

Normal residential/mobile IP behaviour

Retry within the same sticky session; keep the profile

Every run hits a CAPTCHA

Fresh profile each time

Warm one profile and reuse it (§4)

Instant block on a product/detail page

Deep-linked straight to the URL

Go homepage → search → filter → click through (§5)

Session ends around the 30-minute mark

Maximum session duration reached

Split the job across sessions; carry state in the profile (§7)

Cannot create another profile

10-profile limit reached

Reuse existing profiles; a raise is a support request, reviewed case by case (§7)

New sessions queue or fail to start

Concurrent-session limit in use

Cap your worker pool at the limit and queue the rest (§7)

Recording no longer available

Older than 3 days

Download recordings while debugging, before retention expires (§7)

Did this answer your question?