How to Avoid IP Bans When Web Scraping
Most scrapers are not blocked because of their IP address. They are blocked because of request shape, concurrency, and TLS fingerprint — and rotating the IP harder makes it worse.
Loading page content.
Public profile data and logged-in data are entirely different questions, legally and technically. We allow one and prohibit the other, and here is exactly where the line falls.
Dana Whitfield
· updated 9 Aug 2026
We get this question several times a week, usually phrased as a yes/no. It is not a yes/no. There are two different activities here with different legal analyses, different technical difficulty, and different answers from us.
This is not legal advice. It is a description of where our policy line sits and why.
Publicly visible profile data — what a logged-out visitor sees at a public profile URL, including name, headline, current employer, and public post text.
Logged-in data — anything behind authentication: connection lists, full employment history on restricted profiles, InMail, group membership, Sales Navigator result sets, or search beyond the logged-out limits.
Every meaningful distinction follows from which of these you mean.
The case is cited constantly and usually wrongly.
The Ninth Circuit held that scraping publicly available data likely does not violate the CFAA, because data open to the general public is not "without authorization" in the statutory sense. That was a real and durable win for public-web collection.
Then, in 2022, hiQ lost on the other claim: it had breached LinkedIn's User Agreement, having accepted those terms and having used fake accounts and third-party scrapers. The case settled with hiQ under a permanent injunction and effectively out of business.
The correct summary is: scraping public data was not a computer-crime problem; doing it through accounts that had agreed not to was a contract problem, and the contract problem was fatal. Anyone quoting hiQ as blanket permission has read half of it.
Add to that:
Permitted. Accessing publicly available pages as a logged-out visitor, at a rate that does not degrade the service, where the data class is lawful for you to process and you have addressed your own data-protection obligations.
Prohibited under our Acceptable Use Policy. Authenticating with credentials you are not authorised to use. Operating accounts created under false identity for collection. Credential stuffing or any form of access-control circumvention. Bulk harvesting of personal data with no lawful basis. Continuing to collect from any operator that has served you a specific notice to stop.
Those are not aspirational. A report with evidence triggers the enforcement process in the AUP, which starts with a request for explanation and ends, where warranted, with suspension.
Most requests we see have a business goal that a compliant route serves better.
| Goal | Compliant route |
|---|---|
| Enrich CRM records with firmographics | Licensed data vendors with contractual coverage |
| Track job changes in your pipeline | LinkedIn Sales Navigator, or first-party opt-in signals |
| Market or competitor research | Company career pages, filings, press releases, structured open data |
| Recruiting sourcing | LinkedIn Recruiter, or candidate-consented sourcing tools |
| Academic research on labour markets | Institutional data agreements; many are cheaper than expected |
The recurring pattern: the licensed route costs money and removes an entire category of risk, and teams usually discover the numbers are close once they price the engineering time that automated collection actually consumes.
Assume you have done the analysis and concluded a specific, narrow collection is lawful for you. Technically it is still one of the harder public targets.
Expect strong bot management: TLS fingerprinting, aggressive rate limits on logged-out browsing, quick escalation to challenges, and an authentication wall that appears after a small number of profile views from one address. Datacenter address space is filtered heavily.
# Residential exit, session held for one journey, single-digit requests per minute
curl -x http://user-country-us-session-4c1de907ab52:[email protected]:8080 \
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" \
-H "Accept-Language: en-US,en;q=0.9" \
-sS -o /dev/null -w "%{http_code}\n" https://example.com/public-pageThree engineering rules if you proceed:
Public data collection at modest scale, with a real lawful basis, done as a logged-out visitor, is something we support and will not block. Anything requiring an account, a fake identity, or the circumvention of an access control is something we prohibit and will act on. If a proposed project can only work by crossing that line, the project needs a different design, not a different proxy provider.
Most scrapers are not blocked because of their IP address. They are blocked because of request shape, concurrency, and TLS fingerprint — and rotating the IP harder makes it worse.
The legal picture is more nuanced than either extreme claims, and the technical picture is harder than most tutorials admit. Both, honestly.
Teams underestimate bandwidth by three to ten times, almost always for the same four reasons. Here is how to produce a number you can budget against.
Every snippet in this article points at the production gateway. Create an account, take the 50 MB residential trial, and swap in your credentials.
No card required for the trial. Cancel or downgrade at any time.