Proxies for Market Research: Samples That Are Not Just Your Desk

Market research collected from one country is a study of one country that has been labelled global. The problem is not that the data is wrong — each observation is accurate. The problem is that the sample was drawn from wherever the researcher happened to be sitting.

What localisation hides from a single vantage point

Query a category from a London address and you get the UK assortment, UK pricing, UK promotions and UK competitors. Nothing signals that anything is missing. The German assortment, the local brands that dominate it, and the price points they hold are simply absent, and absence does not show up in a spreadsheet.

Four things vary by market in ways that change conclusions:

• Assortment. Which products are listed at all.
• Price and promotion cadence. Discount depth and timing differ by market even within one retailer.
• Competitive set. Local players that never appear in your home market frequently lead in theirs.
• Positioning. The same product marketed to different segments with different claims.

A study that misses the local competitive set is the expensive failure. It produces a confident market-share estimate built on a denominator that omits the actual leader.

Sample deliberately, then hold the sample still

The fix is unglamorous: choose the markets, choose the locations inside them, and keep both fixed across waves.

for cc in de fr it es pl nl; do
  curl -x "http://user-country-$cc:[email protected]:8080" \
    -s "https://retailer.example/category/coffee" \
    -o "assortment-$cc.html"
done

Rotating residential reaches 195 countries with city and ASN targeting at $2.65 per GB, falling to $1.59 at 500 GB. For research the country list should come from the markets in scope rather than the markets that are easy to reach, and it should be written down alongside the results.

Within a large market, add cities. Assortment and price vary by fulfilment region, so a single national sample is itself a convenience sample. Three or five deliberately chosen cities beat one arbitrary one, and the improvement is larger than most researchers expect.

Waves have to be comparable

Research is longitudinal. A wave in March compared against a wave in June is only meaningful if the instrument did not change.

The things that quietly change between waves, in the order they cause trouble:

1. The city or country list, edited without a note.
2. The Accept-Language header, which changes both currency and copy.
3. The time of day, which moves promotional pricing.
4. The device profile, which changes assortment on mobile-first retailers.
5. The proxy tier, which changes which pages soft-block.

Pin all five and record them with the data. Most unexplained movement between waves is instrument drift, and instrument drift is indistinguishable from a real market shift once the configuration has been forgotten.

Language is part of the location

An exit in Warsaw sending Accept-Language: en-US gets a translated storefront, English copy, and sometimes a different currency. That is not the Polish market — it is the Polish market as shown to an anglophone visitor, which is a different research question and rarely the one being asked.

LOCALE = {
    "de": "de-DE,de;q=0.9",
    "fr": "fr-FR,fr;q=0.9",
    "pl": "pl-PL,pl;q=0.9",
}

headers = {"Accept-Language": LOCALE[country]}

Match the header to the exit country unless you are deliberately studying the expatriate experience. If you are, record that as a separate arm rather than mixing it in.

Choosing a tier, and why it is usually not the expensive one

Research volumes are modest compared with production scraping. A category sweep across six markets is thousands of requests, not millions, which changes the economics.

Start on rotating datacenter at $1.20 per GB with a 0.28s median response. Where the target does not weigh ASN reputation, it collects the same pages faster and cheaper. Move to residential for the domains that block it — and for anything where the content itself is personalised by network, since a datacenter address can be treated as a non-consumer and served a generic experience.

That last point is the one exception worth taking seriously in research. In price monitoring a soft-block is a failure you detect and retry. In research, a generic experience served to a datacenter IP is a plausible-looking page that silently is not what a consumer sees, and it enters the dataset as though it were.

Detecting the generic experience

Three checks separate a real localised page from a fallback:

• Currency symbol matches the market. A euro price in a UK sample is a redirect you did not notice.
• Language of the body copy matches the exit country, not just the header you sent.
• Result counts are plausible. A category returning nine products in one market and four hundred in another is usually a filter or a block, not an assortment difference.

Run all three as assertions rather than eyeballing a sample. Research datasets are large enough that manual inspection covers a rounding error of the rows.

Keep the provenance

Every row should carry where it came from: exit country, exit city, UTC timestamp, language header, and the proxy tier used. It costs a few columns and it is the difference between answering "why did Spain move?" from the data and answering it from memory.

For the wider workflow see market research, and how to avoid IP bans when web scraping for the collection mechanics.

A checklist

1. Draw the market list from the research scope, not from convenience.
2. Add cities inside large markets; a national sample is still a convenience sample.
3. Match Accept-Language to the exit country.
4. Pin the city list, language, time of day, device and tier — and record them per row.
5. Assert currency, body language and result count before accepting a page.
6. Prefer datacenter where it works; escalate for personalised content.
7. Treat any change to the instrument as a new wave, and note it.