Proxies for Price Monitoring: Collecting Prices That Are Real

Retailers do not have a price. They have a price per market, per fulfilment region, sometimes per visitor. A monitoring pipeline that ignores that does not collect competitor prices — it collects one arbitrary sample and reports it as a fact.

The price depends on who is asking

Three mechanisms vary what a retailer shows, and they compound:

Fulfilment region. Large retailers price by warehouse catchment. The same URL returns different availability, different delivery promises and frequently different prices in Los Angeles and rural Montana.

Currency and market. Cross-border storefronts redirect or re-price by IP geography before you see anything.

Visitor history. Less common than the folklore suggests, but real on travel and ticketing, where an address that has checked the same route repeatedly is a known signal.

The consequence is that "their US price" is not a number. It is a distribution, and which part of it you sampled depends on which exits your pool happened to use — which changes week to week. Prices appear to move when nothing moved.

Fix the locations, then keep them fixed

Deliberate, repeated locations turn a noisy sample into a time series:

for city in losangeles chicago newyork dallas miami; do
  curl -x "http://user-country-us-city-$city:[email protected]:8080" \
    -s "https://retailer.example/product/12345" \
    -o "price-$city.html"
done

Pick the cities once, from where your own demand actually is, and do not change the list casually. Every change is a discontinuity in the series, and a change made quietly is a discontinuity nobody can explain three months later.

Rotating residential covers city-level targeting across 195 countries at $2.65 per GB, falling to $1.59 at 500 GB. City targeting is included rather than sold as an upgrade, which matters here because location is the measurement.

Do not use the most expensive tier by default

Price monitoring is high-frequency and repetitive, which makes it one of the more expensive workloads to run badly.

Start on rotating datacenter: 240,000 addresses, 0.28s median response, $1.20 per GB down to $0.72 above a terabyte. Plenty of retail sites do not weigh ASN reputation, and against those the cheap tier collects identical data at roughly half the price and twice the speed.

Escalate to residential only for the domains that demonstrably block datacenter exits. The test is one afternoon of work:

dc=$(curl -x http://user-dc-country-us:[email protected]:8080 \
      -s -o /dev/null -w '%{http_code}' "$URL")
res=$(curl -x http://user-country-us:[email protected]:8080 \
      -s -o /dev/null -w '%{http_code}' "$URL")
echo "datacenter=$dc residential=$res"

Run it across your target list and split the list in two. Most pipelines end up with the majority of domains on the cheap tier, and the blended cost per collected price drops accordingly.

Rotate per request, with one exception

Each product check is stateless, so per-request rotation is correct and keeps the per-address request rate low.

The exception is any flow where the price only appears after several steps — add to cart, enter a postcode, reach a delivery estimate. Those steps have to arrive from one address or the retailer's session logic falls apart:

curl -x http://user-country-us-city-chicago-session-cart8841:[email protected]:8080 \
  -s https://retailer.example/cart

Set the session token per logical check, not per run. One token doing four hundred product pages is the concentration pattern rate limiters exist to catch.

Detect soft blocks or your prices become fiction

The dangerous failure is not a 403. It is HTTP 200 with a page that is subtly not the real one: a generic catalogue, a cached version, a challenge interstitial. Parsers handle these by finding no price and recording nothing, or worse, by finding a placeholder and recording that.

Three defences, all cheap:

• Assert the page is the product. Check the SKU or product title in the response before you trust the price. A page that does not contain the identifier you requested is a failure regardless of status code.
• Alert on implausible deltas. A 60% overnight price drop is far more likely to be a parsing error or a different regional catalogue than a real promotion. Quarantine it for review rather than publishing it.
• Watch response size by domain. Challenge pages cluster small. A bimodal size distribution is a soft block you have not noticed.

Bandwidth is where the budget goes

A product page is mostly images. The price is a few hundred bytes of HTML, and you are paying residential rates for megabytes of photography you discard.

await page.route("**/*", lambda route: (
    route.abort()
    if route.request.resource_type in {"image", "media", "font"}
    else route.continue_()
))

Blocking those types typically removes the large majority of page weight. Better still, check whether the retailer exposes a JSON endpoint the page itself calls — many do, and fetching it directly is both cheaper and more stable than parsing markup that changes every quarter. Estimating proxy bandwidth covers the arithmetic.

Sampling schedule matters as much as sampling location

Prices change through the day. A daily 03:00 sample and a daily 18:00 sample tell different stories, and comparing a competitor's morning price to your own evening price produces a gap that does not exist.

Fix the time, fix the timezone, and record both in the row. Then, when someone asks why a series jumped, you can answer from the data instead of guessing.

For the broader workflow see price monitoring.

A checklist

1. Choose a fixed city list from your own demand and treat changes as a schema change.
2. Test every domain against datacenter first; escalate only what fails.
3. Rotate per request; use a session token only for multi-step cart flows.
4. Verify the SKU appears in the response before recording a price.
5. Quarantine implausible deltas instead of publishing them.
6. Block images, fonts and media, or call the retailer's own JSON endpoint.
7. Fix the sampling time and store it with the timezone.