> ## Documentation Index > Fetch the complete documentation index at: https://docs.nodemaven.com/llms.txt > Use this file to discover all available pages before exploring further. # Proxy & Profile Best Practices > How to configure proxies, browser profiles and the CAPTCHA solver in the Scraping Browser so your runs stay fast, consistent and unblocked. How to configure proxies, browser profiles and the CAPTCHA solver so your Scraping Browser runs stay fast, consistent and unblocked. This page is about *choices*, not mechanics - it assumes you can already create a session and connect to it. For endpoints, parameters and code, see the [Scraping Browser API reference](/scraping-browser/api-reference). ## 1. The proxy is your identity, not just a pipe A proxy is not a transport detail. It is **the channel through which every site, API and LLM perceives you**. Before a single line of your script runs, the target has already decided what to show you based on the IP alone: | What the IP determines | Effect on your results | | - | - | | Country / region / city / ISP | Which catalogue, currency, language and prices you see. Google Shopping, marketplaces and LLM answers are all geo-shaped, and results can differ between regions of the same country. | | Network type (residential / mobile) | Whether you are treated as a person or as automation. Datacenter IPs are not supported here - and would not survive these targets anyway. | | IP reputation and history | Whether you get content, a CAPTCHA, a soft block, or degraded/decoy results. | The most common cause of "bad data" is not the scraper. It is a mismatched or unstable IP producing a **different page** than the one you expected. **Rule:** decide the geo you actually want *before* writing selectors. Target the location of the audience whose results you are trying to reproduce. **Verify it before you trust the data.** On the first run of any new configuration, check where you are actually coming out - open an IP-echo page, or watch the session's live view - and confirm the country, region and currency the target is serving you. Most "the data looks wrong" reports are a session that was never in the location it was configured for. ## 2. Choosing the proxy type The Scraping Browser runs on **residential and mobile proxies only**. Datacenter and static/ISP proxies are not supported - they are the first thing a serious anti-bot system filters out, so the browser is built around real-network IPs by design. **Residential (30M+ IPs)** - real IPs on real consumer ISPs. The default for search engines, marketplaces, PLA/Shopping data and anything with serious bot protection. **Mobile (295K+ IPs)** - IPs from mobile carrier networks (4G/5G). The highest trust level available, because carrier-grade NAT means thousands of real users share the same address and blocking it is expensive for the target. The pool is smaller, so reserve it for where it counts: the hardest targets, mobile-first platforms and social apps, or when residential is still getting challenged. Both pools are filtered for low fraud score before an IP ever reaches you, so what you get is a real, clean address that looks like a real user to the target. That filtering is the product. What no provider can remove is the nature of the network itself: these are real devices, and a real device can go offline mid-session. It happens occasionally - the exception, not the rule - so build for it rather than assuming a permanently open connection. The **Sticky** session type absorbs it for you by substituting an equivalent IP (§3). **Rule of thumb:** start on residential. Move to mobile when the target keeps challenging you, or when it is a mobile-first platform. ## 3. Proxy settings Everything in this section lives under **Proxy Settings** in the dashboard. ### Location settings Target the location explicitly - **Country**, then **Region** and **City** when the data is local (prices, availability, local packs). **ISP** and **ZIP targeting** narrow it further when you need to reproduce a very specific user's view. Do not leave geography to the pool if it affects the output. Country and region show a quality badge (High / Medium / Low) indicating pool depth for that location. Prefer High where you can - a thin pool means fewer IPs to rotate through and faster reuse of the same addresses. ### Session type | Mode (dashboard label) | What it does | Use it when | | - | - | - | | **Rotating** | A new IP on every request / every new session. | High-volume, stateless collection where each request stands alone and you *want* to spread across many addresses. | | **Sticky** - *Keep IP as long as possible (Super sticky sessions)* | Holds one IP for the session. If that IP goes offline, the system automatically substitutes another IP matching the **same targeting** (country, region, city) so the session keeps running. | The default for most jobs. You want one stable identity, but you care more about the run finishing than about it being literally the same address end to end. | | **No rotating** | Pinned to one specific IP. If it drops, nothing is substituted - the session waits for that IP to come back rather than silently switching you to another one. | Logged-in accounts, long multi-step flows, and anything where an IP change mid-session would break trust or invalidate a solved challenge. Gives you full manual control: you decide when to switch. | The trade-off is continuity vs. identity. **Sticky** keeps the job alive and absorbs the fact that residential and mobile IPs drop; **No rotating** keeps the identity perfectly intact but hands you the drop to deal with yourself. Rotating per request looks like a hundred different people arriving at the same funnel in the same second. Sticky looks like one person browsing - and your cookies, consent state and any solved CAPTCHA stay valid for the rest of the run. **Rule:** one job = one location = one sticky session. Rotate *between* jobs, not inside them. Use **No rotating** whenever the session carries a logged-in state or a solved challenge. ### Protocol Two options: **HTTP** and **SOCKS5**. **Use SOCKS5 for protected sites.** It is the more reliable choice against targets with serious bot protection - Google in particular - and we have measured noticeably more consistent results with it. HTTP is fine for undefended endpoints and simple fetches. **Rule:** protected target → `SOCKS5`. Make it your default and only drop to HTTP if something specifically requires it. ### IP filter mode Controls how the pool picks an IP for you. | Mode | Optimises for | Use it when | | - | - | - | | **Quality (default)** | Lowest fraud score. Filters the pool down to clean IPs with good reputation. | Leave it here. This is what gets you through anti-bot checks - clean history beats raw speed on any protected target. | | **Quality + Speed** | Clean IPs, then the fastest among them. | You need volume but still care about getting through. A reasonable compromise for large jobs on moderately protected sites. | | **Speed** | Lowest latency. | Undefended endpoints where throughput is the only thing that matters. | | **Max pool size** | The widest possible set of IPs, with the least filtering. | You need maximum IP diversity - very large rotating jobs - and the target does not scrutinise IP reputation. | **Rule:** stay on **Quality (default)** unless you have measured a reason not to. Loosening the filter trades fraud score for speed or pool size, and fraud score is exactly what protected sites check. ### IP version **Mixed IPv4 and IPv6** by default. Switch to **IPv4 only** if the target handles IPv6 badly or serves different content over it - some sites still treat IPv6 ranges as higher risk. ## 4. Browser profiles: solve the CAPTCHA once This is the single biggest speed and success-rate lever in the product. Under **Browser Profile Settings**, the **Browser state** toggle decides everything here: * **One-time** - state is discarded when the session ends. Every run starts as a stranger and pays for the CAPTCHA again. * **Persistent** - state is kept and reused. **This is the setting you want** for any repeated job. A **persistent browser profile** keeps the real Chrome state between sessions - cookies, local storage, history, consent flags and the trust signals a site has already granted you. Consequences: * The first run on a hard target may hit a challenge. You solve it once. * The next run reuses those cookies, so the site recognises a returning visitor and, in most cases, **does not challenge you again**. * Runs get dramatically faster, because you skip the solve entirely instead of paying 1-3 minutes for it every time. * Behaviour becomes reproducible: same profile, same IP, same page. **Pin the profile to the proxy.** A profile carrying cookies issued to a US residential IP that suddenly appears from Germany is a contradiction, and it is exactly the pattern anti-bot systems look for. Bind each profile to one location and keep it on the same (or closest available) IP every time. **Rules:** * Set **Browser state → Persistent**. One-time profiles throw away everything this section is about. * Create one profile per target + geo combination, and reuse it. Do not create a fresh profile per run. * Store the profile ID alongside the proxy config you used with it, and always pair them again. * Never move a warmed profile to a different country. * Warm a profile with a couple of light runs before pointing it at your heaviest job. ### Other browser settings worth knowing | Setting | Recommendation | | - | - | | **Extra Stealth** - managed fingerprint and anti-detection | **On.** Together with the CAPTCHA Solver it does the heavy lifting on bot checks. | | **CAPTCHA Solver** | **On.** See §6 for how long to wait for it. | | **Browser mode** | **Headful (recommended).** Headless is a detection signal on protected sites. | | **Ad Blocker** / **Popup Blocker** | On by default. Blocks ads and unnecessary resources, which speeds up page loads. Turn the Ad Blocker off if a target breaks or you actually need the ad/PLA units to render. | | **Cloudflare Web Bot Auth** | Off by default. Signs your requests for participating Cloudflare sites - turn it on only when you are working with a site that expects an identified bot. | | **Idle timeout** | 5 minutes, fixed. A session with no activity for that long closes - so keep long waits (like a CAPTCHA solve) filled with actual polling (§6). | ## 5. Arrive the way a real user would Never deep-link straight to the page you actually want. Jumping from a search engine - or from nothing at all - directly into a product page, a listing or a detail URL is one of the clearest automation signals a platform has, and it is a common reason for an instant block or challenge on an otherwise perfect setup. A real customer does not materialise on `amazon.com/dp/B0XXXXXXX`. They land on the homepage, search, filter, and click through. Reproduce that path: 1. **Open the homepage first.** Let the site set its cookies, consent state, session and regional context. 2. **Search from the site's own search box** - type the query you actually want, don't rewrite the URL. 3. **Apply filters and sorting in the UI** rather than constructing a query-string that no human would produce. 4. **Click through to the product** from the results list. 5. **Pace it.** Leave roughly **1-3 seconds** between actions and let each page settle before the next click. That is enough to stop looking like a script without burning your session budget - a five-step path costs you well under a minute. The referrer chain, the cookies picked up along the way and the timing between steps all end up in the platform's risk score. Deep-linking discards every one of them. This compounds with §4: the navigation warms the profile, the profile keeps that warmth for the next run, and after a few clean passes the site stops treating you as a first-time stranger. **Rule:** homepage → search → filter → product. Direct-to-product URLs only from a profile that has already earned trust on that domain. ## 6. The CAPTCHA solver: it works, but give it time The built-in solver handles challenges reliably - including Google's `/sorry/` interstitial. The thing to internalise is **it is not instant**. * Typical clear: a few seconds. * Hard challenges: **\~1 to 3 minutes**, occasionally longer. * It generally does get through - the failures we see in the wild are almost always *client-side timeouts firing while the solver was still working*. So the default 30-60 second timeout most scripts ship with is the wrong number here. **Rule:** set the CAPTCHA wait to **at least 240 seconds (3-4 minutes)**. Poll the page for the challenge clearing rather than sleeping a fixed amount, and only then continue. **Poll actively while you wait - don't sleep.** This is the trap: the session closes after 5 minutes of inactivity, and that limit is fixed (§7). A blind `sleep(240)` is 4 minutes of nothing happening, and the session can die right as the solver finishes. Poll the page every few seconds instead - that keeps the session alive *and* lets you continue the moment the challenge clears. The 5-minute idle window is therefore your real ceiling: treat **\~4 minutes as the practical maximum** for a single CAPTCHA wait. And budget the session as a whole - a hard solve can eat 3 of your 30 available minutes. Combine this with §4: with a warmed profile you will usually not reach the challenge at all, which is why the profile is worth setting up. ## 7. Default limits These limits are fixed - they are not settings you can change. Plan your jobs around them: | Limit | Default | | - | - | | Browser profiles | **10** | | Concurrent active sessions | **50** | | Maximum session duration | **30 minutes** | | Idle timeout | **5 minutes** | | Session recording retention | **3 days** | What this means in practice: * **Budget the 30-minute duration.** A hard CAPTCHA can eat 1-3 minutes of it (§6). Size each job to fit inside one session, and continue longer work in a new session on the same persistent profile so it does not start cold. * **10 profiles is a design constraint, not a nuisance.** It maps naturally to one profile per target + geo (§4). Reuse them; don't burn them on one-off runs. * **50 concurrent sessions** is your real parallelism ceiling - size your worker pool to it and queue the rest. * **Watch the 5-minute idle timeout.** A session with nothing happening on it closes. Never park a session between steps, and never block on a bare `sleep()` - poll instead (§6). * **Pull recordings within 3 days.** If you are debugging a failed run, download the recording before it ages out. Need more than this? Write to support with your use case. Limit increases are reviewed case by case rather than granted on request, so explain what you are running and why the default does not fit. ## 8. Recipe: Google Search and Shopping (PLA) scraping Google is the hardest common target, so it makes a good worked example. This configuration reliably pulls clean product data - titles, prices and merchant URLs - from both the inline shopping unit and the Shopping tab, with Extra Stealth and the CAPTCHA Solver doing the work. ### Configuration checklist | # | Setting | Why | | - | - | - | | 1 | **Do not use the in-dashboard Playground.** Run your own script against the CDP API. | The Playground sandbox caps execution at **60 seconds**. Google's CAPTCHA can take longer than that to clear, so a Playground run is killed mid-solve and never finishes. Connecting your own local or server-side script directly to the session's CDP URL has no such limit. | | 2 | **CAPTCHA wait ≥ 240s** | See §6. Google challenges have taken \~1-3 minutes in our runs. | | 3 | **Protocol: `SOCKS5`** (not HTTP) | Noticeably more consistent results against Google specifically. Set it under Proxy Settings when creating the session (§3). | | 4 | **Extra Stealth: ON** and **CAPTCHA Solver: ON** | These two do the heavy lifting on Google's bot checks. | | 5 | **Residential (or mobile) + Sticky + IP filter mode: Quality + the exact country/region/city you want prices for** | Google Shopping output is geo-dependent, and a clean low-fraud-score IP is what gets you past the bot check; see §1 and §3. | | 6 | **Reuse a warmed browser profile** | After the first solve, most subsequent runs skip the CAPTCHA wall entirely. | | 7 | **Poll while waiting for the CAPTCHA, never `sleep()`** | The session closes after 5 idle minutes, so a blind sleep can kill it mid-solve (§6). | | 8 | **Turn the Ad Blocker OFF** | PLA units *are* ads. The Ad Blocker is on by default and can strip the very Shopping results you came for - if your extraction returns zero products on a page that clearly has them, check this first. | One practical note on the flow: if the regular search returns no inline shopping unit, retry the query on the Shopping tab (`&tbm=shop`). Expect the `/sorry/` wall on either step - handle it the same way both times. ## 9. Quick reference ```yaml theme={null} proxy: type: residential # residential or mobile - the only supported types protocol: socks5 # http | socks5 - socks5 for protected sites ip_filter_mode: quality # quality (default) | quality+speed | speed | max-pool-size ip_version: mixed # mixed ipv4+ipv6 | ipv4-only country: us # always pin explicitly region: # state/province - results differ within a country city: # pin when prices/availability are local isp: # narrow further if you need a specific provider's view session_type: sticky # rotating | sticky | no-rotating # no-rotating for logged-in / long flows browser: profile: # one profile per target+geo, bound to the proxy above browser_state: persistent # NOT one-time - this is what keeps the cookies browser_mode: headful extra_stealth: true captcha_solver: true navigation: path: homepage -> search -> filter -> product # never deep-link to a detail URL pacing: let each page settle before the next click limits: # fixed - support reviews increases case by case browser_profiles: 10 concurrent_sessions: 50 session_duration_minutes: 30 idle_timeout_minutes: 5 recording_retention_days: 3 timeouts: captcha_wait_seconds: 240 # never 30-60 execution: run-your-own-script # Playground caps at 60s ``` ## 10. Troubleshooting | Symptom | Likely cause | Fix | | - | - | - | | Wrong prices, currency or catalogue | IP is in the wrong geo | Pin country, region and city explicitly (§1, §3) | | Results change between identical runs | Rotating IP or a new profile each run | Sticky session + reuse one profile (§3, §4) | | IP silently changed mid-session | Smart rotation replaced a dropped IP | Switch to sticky with no replacement if the session must keep one address (§3) | | Session stalls after an IP drops | No-replacement stickiness waiting for the IP | Expected behaviour - wait, or switch proxy manually (§3) | | Script dies "on the CAPTCHA" | Client timeout fired mid-solve | Raise the wait to ≥240s (§6) | | Run always killed around 60s | Running in the Playground | Connect your own script to the CDP URL (§8.1) | | Google blocks persistently | HTTP protocol, or stealth/solver off | Switch to SOCKS5, enable both (§8.3-8.4) | | Blocked despite a clean setup | IP filter mode loosened past Quality | Return to **Quality (default)** for a low fraud score (§3) | | Profile forgets cookies between runs | Browser state left on One-time | Switch to **Persistent** (§4) | | Session closed while nothing was happening | 5-minute idle timeout (fixed) | Poll instead of sleeping; keep the session busy (§6) | | Zero products extracted from a page that has them | Ad Blocker stripped the PLA units | Turn the Ad Blocker off for Shopping scraping (§8) | | Job doesn't fit in one session | 30-minute maximum | Continue in a new session on the same persistent profile (§7) | | Connection drops mid-run | Normal residential/mobile IP behaviour | Retry within the same sticky session; keep the profile | | Every run hits a CAPTCHA | Fresh profile each time | Warm one profile and reuse it (§4) | | Instant block on a product/detail page | Deep-linked straight to the URL | Go homepage → search → filter → click through (§5) | | Session ends around the 30-minute mark | Maximum session duration reached | Split the job across sessions; carry state in the profile (§7) | | Cannot create another profile | 10-profile limit reached | Reuse existing profiles; a raise is a support request, reviewed case by case (§7) | | New sessions queue or fail to start | Concurrent-session limit in use | Cap your worker pool at the limit and queue the rest (§7) | | Recording no longer available | Older than 3 days | Download recordings while debugging, before retention expires (§7) | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.