Can I Scrape Job Data From Indeed's Website Legally and Effectively?

Short answer: sometimes, with real constraints, and not in the way most tutorials describe. This post separates three questions that get collapsed into one — what the law says, what Indeed's terms say, and what is technically possible — because the answers differ and only one of them is about code.

Nothing here is legal advice. If the programme is commercially significant, have a lawyer in your jurisdiction look at it before it ships.

The legal picture

Public data and computer-misuse law. In the United States, Van Buren (2021) and the Ninth Circuit's hiQ v. LinkedIn line narrowed the Computer Fraud and Abuse Act considerably: accessing a public website you are permitted to view is generally not "unauthorised access" under the CFAA merely because the operator objects. That is a meaningful shift from a decade ago. It is also narrower than the headlines suggested — hiQ ultimately lost on breach of contract, which is the point people skip.

Contract is the live risk, not the CFAA. Indeed's Terms of Service prohibit automated access, scraping, and the use of extraction tools. If you accepted those terms — by creating an account, and in many jurisdictions by continuing to use a site with conspicuous browsewrap notice — you are in a contractual relationship, and breach of that contract is actionable independently of any criminal statute. This is the mechanism that has produced most real outcomes in scraping disputes.

Copyright. A single job posting's descriptive text is authored content and can be protected. Facts are not: job title, company, location, posting date, and salary range are facts, and a database of facts is generally outside copyright in the US, though the EU's sui generis database right takes a different view of substantial extraction. Reproducing full posting text verbatim is the risky end; storing normalised fields is the safer end.

Personal data. Job listings occasionally include a named recruiter and a direct email. That is personal data under GDPR and comparable regimes, and collecting it triggers obligations — lawful basis, purpose limitation, and Article 14 notice to a person you never interacted with, which is genuinely difficult to satisfy at scale. The pragmatic answer is to filter contact details out at the parser, before they reach storage.

Jurisdiction. German and French courts have been notably less permissive than US courts on automated collection. If you or your users are in the EU, US case law is not the relevant analysis.

What Indeed offers instead

Before building anything, price the sanctioned route.

The third row is under-considered. Most job data that appears on Indeed also appears on the employer's own careers page, published deliberately for maximum distribution, frequently in structured JobPosting JSON-LD, and usually without an aggregator's anti-automation terms. If your goal is a labour-market dataset rather than a mirror of one platform, going to the source is often cheaper, cleaner, and less contested.

The technical picture

Assume you have concluded the collection is lawful for your specific case and scope. It is still hard.

Large job boards run mature bot management: TLS/JA4 fingerprinting, challenge interstitials, per-subnet rate limits, and behavioural scoring across a session. Datacenter address space is filtered aggressively. Naive request loops die within minutes.

What collection actually requires:

• Residential or mobile exits, geo-matched to the market. Job listings are heavily localised; a US exit will not give you the German result set.
• Sticky sessions per search journey. Search-result pagination carries state. An address that changes between page 2 and page 3 invalidates the sequence and often triggers a challenge.
• A real browser for anything challenge-gated, driven with a coherent fingerprint. Header spoofing alone stopped working years ago.
• Very low request rates. Single-digit requests per minute per session is the honest range for this class of target, not per second.

import httpx

def search_session(country: str, session_id: str) -> httpx.Client:
    """One client per search journey; the session segment holds the exit IP."""
    proxy = (
        f"http://user-country-{country}-session-{session_id}"
        f":[email protected]:8080"
    )
    return httpx.Client(
        proxy=proxy,
        timeout=45.0,
        follow_redirects=True,
        headers={
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                          "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
            "Accept-Language": "en-US,en;q=0.9",
        },
    )

Structured data, where it is present, is worth checking first — it is stable across redesigns in a way that CSS selectors are not:

import json
from selectolax.parser import HTMLParser

def job_postings(html: str) -> list[dict]:
    """Prefer schema.org JobPosting blocks over CSS selectors; they survive redesigns."""
    out: list[dict] = []
    for node in HTMLParser(html).css('script[type="application/ld+json"]'):
        try:
            data = json.loads(node.text())
        except json.JSONDecodeError:
            continue
        items = data if isinstance(data, list) else [data]
        out.extend(i for i in items if i.get("@type") == "JobPosting")
    return out

What we will and will not support

FleetProxy sells network access; what you do across it is governed by our Acceptable Use Policy and by the law where you operate. We do not permit circumventing authentication, collecting data you have been specifically served notice to stop collecting, or harvesting personal data without a lawful basis. We do not gate ordinary public-web collection either, and we do not pretend that a proxy makes a contractual restriction disappear.

If the goal is labour-market intelligence, the highest-quality dataset we see customers build is assembled from employer career pages and licensed feeds, with aggregator data as a sparse supplement. It is more engineering up front and dramatically less operational and legal friction afterwards.