Design Google News

Gather articles from thousands of publishers, group stories about the same event, and show each reader a fresh, personalized front page.

We are building a news aggregator: crawl articles from ~50k publishers, cluster articles about the same event into stories, and serve each reader a fresh, personalized front page plus per-locale top stories and topic streams. Clicks go to the publisher's site; we never host the full article.

Functional requirements

• Ingest new articles from publisher feeds (RSS/Atom, sitemaps, push APIs) within minutes of publication.
• Group articles about the same event into a story cluster with a representative headline and several sources.
• Personalized "For you" feed per signed-in reader; anonymous readers get locale top stories.
• Top stories per locale (country + language), and topic streams (Technology, Sports, a named entity like "OpenAI").
• Readers follow/unfollow topics and sources, and hide a source ("fewer stories like this").
• Clicks open the original article through a redirect we log for ranking.

Non-functional requirements

• Freshness: p95 under 5 minutes from publish to appearing in a cluster; breaking stories surface in top stories within 10 minutes.
• Feed read p95 under 200 ms server-side, 99.95% available; a stale feed is better than an error.
• Read-heavy: reads outnumber article writes by ~10,000×.
• Respect publisher licensing: store headline, snippet, image thumbnail and metadata only.
• Global: dozens of locales, each with its own ranking and top stories.

Back-of-the-envelope estimation

• 50k publishers, polled every 1–15 min by tier → ~20k fetches/min ≈ 350 fetches/s; most return 304 Not Modified.
• New articles: ~2M/day ≈ 25/s average, 200/s peak during breaking news.
• Stored per article: metadata + snippet + embedding (256 floats × 4 B = 1 KB) ≈ 3 KB → 6 GB/day, ~2 TB/year hot plus thumbnails in blob storage.
• Readers: 300M DAU × 5 feed loads = 1.5B reads/day ≈ 17k QPS average, 60k QPS peak.
• Clicks: 300M × 4 = 1.2B/day ≈ 14k/s, a stream for ranking signals.
• Candidate pool: articles from the last 48 h ≈ 4M, in ~400k clusters; small enough to keep per-locale indexes in memory.

GET  /v1/feed?cursor=&limit=20          (auth) → {items: [StoryCard], next_cursor, generated_at}
GET  /v1/top?locale=en-US&limit=30      → {items: [StoryCard], as_of}
GET  /v1/topics/{topic_id}?cursor=      → {topic: {id, name}, items: [StoryCard], next_cursor}
GET  /v1/stories/{cluster_id}           → {headline, articles: [ArticleRef], timeline, related_topics}
PUT  /v1/me/follows/{kind}/{id}         kind ∈ topic|source → 204
DELETE /v1/me/follows/{kind}/{id}       → 204
POST /v1/me/hide  {source_id | cluster_id} → 204
GET  /r/{token}                          → 302 Location: publisher URL

StoryCard = {cluster_id, headline, top_article: ArticleRef, more_sources: int, updated_at, reason};
ArticleRef = {article_id, title, snippet, publisher, published_at, thumb_url, click_url}.
click_url is /r/{token} where the token is a signed blob of {article_id, user_hash, surface, position, ts}, so the redirect needs no lookup. The cursor encodes the ranked snapshot id plus offset, so paging does not reshuffle.

Search for the archive (older than 48 h) goes to a separate inverted-index service; it is out of scope for the feed.

flowchart LR
  P[Publisher feeds] --> SCH[Fetch scheduler]
  SCH --> F[Fetchers]
  F --> Q1[(Kafka raw articles)]
  Q1 --> EX[Extract and enrich]
  EX --> DD[Dedup and cluster]
  DD --> ART[(Article and cluster store)]
  DD --> IDX[Per locale candidate index]
  U[Reader] --> API[Feed API]
  API --> PROF[(User profile)]
  API --> IDX
  API --> RANK[Ranker]
  TOP[Top stories job] --> SNAP[(Top stories snapshot)]
  API --> SNAP
  U --> RD[Click redirect]
  RD --> Q2[(Kafka clicks)]
  Q2 --> AGG[Stream aggregators]
  AGG --> IDX
  AGG --> TOP
  Q2 --> PROF

1. Ingestion. The scheduler picks publishers whose next poll time has passed and hands URLs to fetchers, which send conditional GETs (If-None-Match, If-Modified-Since). New entries go to Kafka keyed by publisher. The enrich stage fetches the page once, extracts title, snippet, canonical URL, lead image, language, and runs entity tagging and an embedding model. Dedup drops copies; clustering assigns a cluster_id. The article lands in the store and in the candidate index for its locale within ~2–4 minutes of publish.
2. Personalized feed read. The API reads the user profile (one KV get, cached), pulls ~1,000 candidates from the locale index (followed topics, followed sources, embedding ANN, trending), ranks them with a lightweight model, applies diversity rules, and returns 20 cards with a cursor over a cached ranked list.
3. Top stories and click-through. A job per locale recomputes top stories every 1–2 minutes from cluster velocity, source count and authority; the result is a snapshot served from Redis and the CDN. A click hits /r/{token}, which verifies the HMAC, emits an event to Kafka and returns 302. Stream aggregators turn clicks into per-cluster CTR and velocity, which flow back into the index, the top-stories job and the user's recent-clicks list.

Operational concerns that shape the design:

• Degraded modes. If the ranker times out, return candidates sorted by the fallback score (freshness × authority). If the profile store is down, serve locale top stories with a "Top stories" label. If ingestion stalls, keep serving the existing index; the feed shows older but valid stories. Each step sheds load before it errors.
• Multi-locale. Each locale (en-US, en-GB, de-DE, ...) has its own candidate index, authority scores and top-stories job. A cluster can span locales when the same event is covered in several languages; cross-lingual embeddings let clusters link, but ranking stays per locale.
• Licensing boundary. The enrich stage reads the full page in memory to extract features, then discards the body. Only headline, a snippet (≤ 200 chars, or none where the license forbids it), thumbnail and derived features are stored. Paywalled and opted-out publishers are flagged in the publishers table and honored at fetch and display.
• Observability. Per-publisher freshness lag (publish time to index time) as a histogram; alert when p95 exceeds 5 min or a top-tier publisher has zero new articles for 2× its usual gap. Feed staleness: age of the newest card in served feeds and top-stories snapshot age per locale. Cluster health: duplicates in top 10, mean cluster size.

Fan-in ingestion at 50k-publisher scale. Polling everyone every minute wastes 70k requests/min on unchanged feeds and gets us rate-limited. Instead:

• Adaptive interval per publisher: interval = clamp(1 min, 60 min, 0.5 × median gap between recent posts); wire services and major outlets land at 1 min, small blogs at 30–60.
• WebSub/push where publishers support it; polling becomes a backstop.
• Conditional GETs keep 80–90% of polls at a 304 with no body.
• Politeness: per-host concurrency of 1–2 and a token bucket per domain; fetchers are sharded by host hash so one shard owns each domain's budget.
• The scheduler is a priority queue keyed by next_poll_at, partitioned across 32 workers by publisher id; a missed heartbeat reassigns a partition.
• The enrich stage is the expensive part (page fetch + model inference ≈ 300 ms); at 200 articles/s peak that is ~60 concurrent workers, autoscaled on Kafka lag.

Clustering and syndicated-wire dedup. Two different problems:

• Duplicates: an AP or Reuters piece republished by 300 local papers. Canonical URL catches some; the rest need near-duplicate text. Compute a 64-bit SimHash of the body shingles; Hamming distance ≤ 3 means same text. Keep the earliest or highest-authority copy as primary and attach the others as "also published by". This stops one wire story from filling the feed.
• Same event, different text: assign each new article to a cluster by ANN over centroids of clusters active in the last 48 h, filtered by locale and shared entities. Join if cosine ≥ 0.82 and at least one shared named entity; else start a new cluster. Centroids update incrementally. A batch job every 30 min re-clusters the last 48 h to merge clusters that grew toward each other and split ones that drifted; cluster ids are kept stable by mapping new clusters to the old id with the largest overlap.

Read path under p95 200 ms. Budget: profile 10 ms, candidate retrieval 30 ms, feature fetch 30 ms, ranking 60 ms, assembly 20 ms, slack 50 ms.

• Candidate generation is a fan-out to sources in parallel, each with a 25 ms deadline: followed topics (inverted list, top 200 by recency), followed sources, embedding ANN on the user vector (top 300), locale trending (top 200), local news by coarse geo. Union and dedupe to ~1,000 clusters.
• Ranking is two-stage: a linear or small GBDT model scores 1,000 clusters on features already in the index (freshness decay, source count, velocity, CTR, topic affinity, source authority) in ~10 ms; a heavier model reranks the top 100.
• Diversity: at most 2 cards per topic in the first 10, no publisher twice in a row, at least one top story.
• The ranked list (200 cluster ids) is cached per user for 5 minutes, so paging and back-navigation are cheap reads. A cold miss costs the full path; ~70% of loads hit the cache.

Hot-publisher and trending detection from partitioned click streams. 14k clicks/s, with breaking stories skewed onto a few clusters. Kafka partitions by article_id; each aggregator keeps a 1-minute and 15-minute sliding window per article (count-min sketch for the long tail, exact counters for the top 10k). Every 10 s each partition emits its local top-K (K = 500) by velocity to a merge stage, which rolls articles up to clusters and publishers and computes the global and per-locale top-K. Merging local top-Ks can miss an item that is moderately hot everywhere, so each partition sends K × 2 and the merger sums; the error is acceptable for ranking. A sudden spike for one publisher (e.g. 20× its baseline) flags a breaking story and also tells the fetch scheduler to poll that publisher every minute for the next hour.

Where the interest profile lives. The profile has a slow part (topic affinities and user embedding, updated by an hourly batch job from 90 days of clicks) and a fast part (follows, hides, last 200 clicks). Both live in one KV row keyed by user_id, ~4 KB. A streaming job appends clicks to the fast part within seconds, so a click on a story shifts the next refresh. Feed servers cache profiles in a local LRU for 60 s, and the user's region holds the primary copy with async replicas elsewhere. Explicit actions (follow, hide) write through and invalidate the cached ranked list, so the next load reflects them immediately. Keeping the profile server-side, not on the device, lets ranking use it across web and app and keeps the model features consistent.

• Adaptive polling saves fetch budget but can add minutes of lag for a quiet publisher that suddenly breaks news; push and the click-spike trigger cover most of that gap.
• Online clustering gives seconds of freshness but fragments early stories; the batch repair fixes that at the cost of clusters occasionally merging under a reader's eyes.
• Caching the ranked list for 5 minutes cuts ranking load ~3× but means a breaking story may not appear until refresh; top-stories cards are injected live to compensate.
• Storing only snippets keeps us within license terms but means we cannot re-extract features from old articles without refetching.
• A server-side profile gives cross-device consistency and better ranking, and gives up some privacy and adds a store on the read path.
• Merging per-partition top-K is approximate; exact global counts would need a single hot aggregator.

Mid-level (L4)

• Clear pipeline from fetch to store to feed, with a working API and data model.
• Recognizes clustering and dedup as distinct steps and proposes a reasonable method for each.
• Caches top stories per locale and uses a redirect to log clicks.

Senior (L5)

• Adaptive polling with conditional GETs and per-domain politeness, sized against the 50k publishers.
• Candidate generation plus two-stage ranking within a stated latency budget, with diversity rules.
• Handles syndicated wire copies, cluster drift, and the profile's fast/slow split.

Staff (L6+)

• Ties click streams back into ingestion (spike-driven polling) and ranking, and explains the top-K merge error.
• Defines degraded modes and freshness SLOs with concrete alerts.
• Draws the licensing and privacy boundaries and per-locale ranking, and states what each design choice costs.