Loading page content.
Loading page content.
Pricing intelligence
Repricing on stale or personalised data is worse than not repricing at all. A monitoring fleet needs the price a shopper in that market sees, on the cadence your pricing team actually acts on.
The problem
Retail pricing stopped being a single number years ago. The figure rendered on a product page is a function of the visitor's country, the currency and tax rules that follow from it, the delivery postcode, the device class, whether the session carries loyalty cookies, and increasingly whether the visitor looks like a returning customer or a first-time one. Poll from a single office IP and you collect one narrow slice of that surface and call it the market price.
The second failure is cadence. A pricing team that reprices at 09:00 needs the competitive set as it stood at 08:00, not as it stood when the crawl last finished at 22:00 the previous day. Meeting a schedule across tens of thousands of SKUs means the fleet has to sustain throughput, and throughput from one address is exactly what triggers rate limiting.
The third is contamination. Repeated visits from one identity train personalisation engines. After a few hundred visits the prices you are shown are a response to your own behaviour, and the dataset is measuring the observer.
Why proxies
Price monitoring is a measurement problem, and a proxy is the instrument that fixes the measurement conditions. Pinning each observation to a country — and to a city where delivery pricing varies within a country — is what makes two observations comparable. Without it you are comparing a Frankfurt price to a Chicago price and calling the delta a competitive move.
Rotation solves contamination as well as rate limiting. A fresh exit per SKU gives every observation the profile of a first-time visitor, which is the closest thing to a neutral baseline a public catalogue offers. Where a target genuinely requires a session — a cart that must be built to reveal a delivered price — a sticky session covers exactly that flow and is dropped afterwards.
Recommended product
Datacenter throughput with per-request rotation, billed per GB
from $2.65/GB
Marketplaces and large retailers that score ASN reputation will serve datacenter traffic a degraded page or none at all. Escalate those hostnames to residential rather than escalating the whole fleet.
from $3.50/IP/wk
Where a competitor's portal requires an authenticated trade account, a stable ISP address per account is the right shape — rotation would look like credential sharing.
Worked example
Country targeting is set in the proxy username, which means the market is a parameter of the request rather than a property of the machine. The same code collects four comparable observations by varying one segment of the credential.
import httpx
USER = "fp_8s2k4d19"
PASSWORD = "Xk7mQ2pTz9vRn4Ls"
MARKETS = ["us", "gb", "de", "au"]
URL = "https://example-retailer.com/p/12345"
def proxy_for(country: str) -> str:
# The targeting segment lives in the username, so one credential covers
# every market and nothing about the host machine has to change.
return f"http://{USER}-country-{country}:{PASSWORD}@gate.fleetproxy.com:8080"
observations = {}
for country in MARKETS:
with httpx.Client(
proxy=proxy_for(country),
timeout=30.0,
headers={"Accept-Language": "en"},
follow_redirects=True,
) as client:
response = client.get(URL)
response.raise_for_status()
# Record the currency alongside the amount. A number without its
# currency is the most common corrupt row in a pricing warehouse.
observations[country] = parse_offer(response.text)
for country, offer in observations.items():
print(country, offer["currency"], offer["amount"], offer["in_stock"])Replace the credential with the one in your dashboard. The gateway host, port and username format are the same across every product.
Pitfalls
Each of these is common, cheap to fix, and expensive to leave in place. They are listed in roughly the order teams hit them.
Strike-through prices, member prices, and cart-only discounts are different fields with different meanings. Capture each separately and keep the raw markup fragment for audit; collapsing them into one number at extraction time destroys information you cannot recover without re-crawling.
A synchronous burst across 50,000 SKUs is both the peak that triggers rate limiting and the shape that makes the sweep look automated. Spread the sweep across its window with jitter. The pricing team cannot act on a two-minute-fresher number anyway.
An out-of-stock competitor is not undercutting you at any price, and repricing against their listed number is a straight margin loss. Availability belongs in the same row as the price and should gate any automated response.
A pipeline that silently carries yesterday's value forward when today's fetch fails will hold a stale price through an entire competitor promotion. Mark the observation missing, alert on the missing rate per hostname, and let the repricing rule decide what to do with a gap.
Questions
Start on 50MB of free residential bandwidth, measure your own targets, and scale into volume tiers that step the rate down as the job grows. Unused bandwidth never expires.
No card required for the trial. Cancel or downgrade at any time.