Design a price tracking service

Source: https://www.hack2hire.com/question-bank/questions/system-design/69e1b3ad95e80ae213e5d46e
Company: Meta
Difficulty: Medium

Problem statement

Track the prices of millions of products across stores and alert users when a product they watch drops below their target price.

Requirements

We are building a price tracker in the style of camelcamelcamel or Keepa: a browser extension and website that show a product's price history chart and alert users when a watched product drops below their target price. Prices come from our own crawlers plus prices the extension reports from pages users are already viewing.

Functional requirements

• Users watch a product (by URL or from the extension) with a target price; we notify them (email, push) when the price falls to or below it.
• Show a price history chart for any tracked product: 30 days, 1 year, all time, with min/max/current.
• The extension shows the chart inline on the store's product page and lets users watch with one click.
• Support many stores (Amazon, Walmart, Best Buy, smaller shops); one canonical product can map to listings on several stores.
• Users manage their watches: list, edit target, delete; see why an alert fired.

Non-functional requirements

• Freshness by demand: heavily watched products refreshed every ~15–60 min, long-tail products daily or weekly.
• Alert latency: notification within 5 minutes of a detected drop; at-least-once delivery, never two alerts for the same drop.
• Chart reads are fast: p99 under 100 ms, since the extension renders on every product page view.
• Be a polite crawler: respect per-store rate limits; one store's blocking must not stall others.
• Ingestion tolerates bad data: a fake or misparsed price from an extension must not trigger mass alerts.

Back-of-the-envelope estimation

• 500M tracked listings, 20M users, 100M active watches (avg 5 per user).
• Crawl budget: 10% hot (50M) at hourly ≈ 14K fetches/s; the remaining 450M at weekly ≈ 750/s. Total ~15K fetches/s; at ~200 KB/page that is ~3 GB/s of fetch bandwidth, so parse only structured data (JSON-LD, APIs) where possible.
• Extension reports: 20M users × ~10 product views/day = 200M reports/day ≈ 2.3K/s avg, ~10K/s peak. These cost nothing to fetch and cover what people actually look at.
• Price changes are rare: ~2% of observations differ from the last price. Store change points only: 500M listings × ~3 changes/week × 52 weeks × ~24 B ≈ 2 TB/year, versus ~50 TB/year if every observation were stored.
• Chart reads: 200M extension page views/day ≈ 2.3K/s avg, ~10K/s peak, mostly cache hits.
• Alerts: a few million drops/day cross some user's target; fan-out per drop is 1 to ~100K (a popular TV on Black Friday).

API design

Public (website and extension), all behind auth except chart reads:

• GET /v1/products/lookup?url=<store url> → {product_id, listing_id, store, title, current_price, currency}. Canonicalizes the URL (strip tracking params, extract ASIN/SKU).
• GET /v1/listings/{listing_id}/history?range=1y&resolution=auto → {currency, points: [[ts, price_cents], ...], min, max, current, last_checked_at}. Server downsamples to ≤ 500 points. Cache-Control: max-age=300, ETag on last change.
• POST /v1/watches {listing_id | product_id, target_price_cents, channels: ["email","push"]} → {watch_id, status: "active"}. Watching a product (not a listing) alerts on any store.
• GET /v1/watches?cursor=... / PATCH /v1/watches/{id} / DELETE /v1/watches/{id}.
• POST /v1/observations (extension) {listing_url, price_cents, currency, in_stock, page_signature, observed_at} → 202. Rate limited per install and per user.

Internal:

• Scheduler.Enqueue(listing_id, priority) → crawl task on a per-store queue.
• Kafka topics: observations (raw), price-changes (validated change points), alerts (per-user notifications).

Data model

The watch index is a projection of watches, kept in sync via CDC. Partitioning by (listing_id, year) caps partition size; a listing that changes hourly writes ~9K rows/year.

High-level design

flowchart LR
  EXT[Browser extension] --> API[API gateway]
  WEB[Web app] --> API
  API --> HS[History service]
  API --> WS[Watch service]
  API --> OBS[Observation intake]
  SCH[Crawl scheduler] --> Q[Per store crawl queues]
  Q --> CR[Crawler fleet]
  CR --> K1[(Kafka observations)]
  OBS --> K1
  K1 --> VAL[Validator and change detector]
  VAL --> PH[(Price history store)]
  VAL --> K2[(Kafka price changes)]
  K2 --> MAT[Alert matcher]
  MAT --> WI[(Watch index)]
  MAT --> K3[(Kafka alerts)]
  K3 --> NOT[Notification service]
  HS --> RC[(Redis cache)]
  HS --> PH
  WS --> DB[(Watches DB)]
  DB -. CDC .-> WI

1. Collection. The scheduler pops due listings from each store's sorted set at that store's allowed rate and enqueues crawl tasks. Crawlers fetch, parse with a store-specific extractor, and publish {listing_id, price, in_stock, source: crawler} to observations. The extension posts the same shape through Observation intake.
2. Validation and change detection. Consumers partitioned by listing_id check each observation (schema, currency, sanity bounds, trust), compare with last:{listing_id}, and drop unchanged prices after bumping last_checked_at. A real change is written to the history store and published to price-changes.
3. Alert matching. The matcher range-scans the watch index for listing_id where target_cents ≥ new_price and last_alerted_price is null or higher, and emits one alert per watch to alerts.
4. Delivery. The notification service dedupes on (watch_id, price_change_id), renders, and sends email/push with retries; it then records last_alerted_price on the watch.
5. History serving. The extension calls lookup then history. The history service serves from Redis; on a miss it reads the daily rollup plus recent raw points, downsamples, caches, and returns. Change events invalidate hist:{listing_id}:*.

Deep dives

Crawl scheduling at scale. A listing's crawl interval comes from its demand: interval = clamp(base / (1 + log(watch_count + views_7d)), 15 min, 7 days), shortened further for listings whose price moved recently or during sales events. Each store has its own token bucket (e.g. 50 req/s for a large store, 1 req/s for a small shop) and its own queue, so a slow or blocking store backs up only itself. Extension observations count as free crawls: if a user's view reported the price 10 minutes ago, push the listing's next crawl out. That covers much of the hot set and cuts crawl load ~30–40%. Discovery of new listings comes from extension lookups of unknown URLs, user watches, and store sitemaps and category pages crawled at low priority. Product matching across stores uses UPC/GTIN when present, then brand + model number, with fuzzy title match only as a suggestion that needs confidence above a threshold.

Event-driven alert fan-out. Never scan all watches per price change. The watch index partitioned by listing_id and sorted by target makes matching a single range scan that touches only triggered watches. For huge fan-outs (100K watchers), the matcher pages through the partition in 1K chunks and emits batched alert events, and the notification service throttles per channel provider. Idempotency: the alert ID is hash(watch_id, price_change_id); the notifier keeps a 7-day dedupe set, so Kafka redelivery does not double-send. Re-alert policy: fire once when crossing below target; re-arm only if the price goes back above target and drops again, or drops a further 5%. Price flaps (sale toggling hourly) are damped by requiring the low price to persist for two observations or 30 minutes before alerting, unless two independent sources agree.

Fast history queries for charts. Storing only change points makes a year of a busy listing ~9K rows and a typical one under 200. A nightly job writes daily rollups (open/min/max/close) so "all time" reads a few thousand rows. The history service picks resolution by range (raw for 30 days, daily for 1 year and up) and downsamples to ≤ 500 points with LTTB so the chart keeps its spikes. The result is cached in Redis per (listing, range) with a 1-hour TTL and explicit invalidation on change; with a skewed popularity distribution, hit rate is above 95%, so Cassandra serves under 500 reads/s. A CDN in front of the history endpoint with max-age=300 absorbs the hottest products.

Untrusted extension data. The extension's reports are free coverage but anyone can forge them. Defenses: sign requests with a per-install key and rate-limit per install, user and IP; reject prices outside ±70% of the trailing 30-day median as unconfirmed; mark extension-sourced changes as pending and only alert once confirmed by a crawler fetch (scheduled immediately at high priority) or by N=3 distinct trusted installs within an hour. Each install gets a trust score that rises when its reports match crawler results and falls when they don't; low-trust reports are only used as hints to schedule a crawl. Region and currency come from the page signature, so a UK price never lands on a US listing.

Politeness and anti-scraping. Honour robots.txt and per-store crawl delays, identify the crawler, and prefer APIs and affiliate feeds. Spread requests over time instead of bursts. Detect blocks (CAPTCHA pages, 403/429, prices missing from a parse) and back off exponentially per store, flagging the extractor for review when the parse failure rate passes 5%. A parse that yields a price 90% below the last price is treated as a likely extractor bug, not a sale, until a second fetch confirms it.

Freshness monitoring. Track, per store and per demand tier, the p50/p95 of now − last_checked_at and the crawl success rate. Alert when hot-tier p95 staleness passes 2 hours or a store's success rate drops below 90%. The chart shows "last checked" so users can judge stale data.

Trade-offs

• Change-point storage cuts storage ~25× but loses "we checked and it was unchanged" detail; last_checked_at on the listing restores freshness information.
• Demand-based crawl frequency spends budget where users look, but long-tail listings can miss short flash sales.
• Using extension data gives near-real-time coverage of popular products at zero fetch cost, but requires a confirmation step that adds minutes of alert latency for extension-detected drops.
• A separate watch index partitioned by listing makes fan-out cheap but is a second copy kept in sync by CDC, so a new watch can miss a drop in the seconds before it propagates.
• Damping flapping prices avoids spammy alerts but delays alerts by up to 30 minutes for genuine short sales.
• Cassandra for history gives cheap append writes and partition-local reads but no ad-hoc analytics; export to a warehouse for "biggest drops today" pages.

Interviewer expectations

Mid-level (L4)

• Clear separation of crawling, price storage, watches, and notifications.
• A working alert path: detect a change, find watches whose target is met, send a notification.
• A reasonable history schema keyed by listing and time.

Senior (L5)

• Stores change points only and sizes storage, with rollups and downsampling for chart reads.
• Inverts the alert query with a watch index keyed by listing and sorted by target; handles idempotent, at-least-once delivery and re-arm rules.
• Demand-driven crawl scheduling with per-store rate limits and isolation.

Staff (L6+)

• Treats extension data as untrusted: trust scores, confirmation crawls, outlier rejection, and what that costs in latency.
• Thinks through flapping prices, mega fan-outs on sale days, extractor breakage, and staleness SLOs with concrete detection.
• Cross-store product matching and its error modes, and where the design breaks first at 10× scale.