0

Design a feed recommendation system

by Frontier Interview· 1 day ago· ⑂ 0 remixes

Question: Design a feed recommendation system

Companies: Perplexity, Figma

Difficulty: Hard

Problem statement

Rank the items in each user's home feed so they engage more, choosing from millions of candidates within a couple of hundred milliseconds.

Problem framing

  • Business goal: more users come back to the home feed and find it worth their time. The north-star is daily engaged users and 7-day retention, not raw clicks. A feed that farms clicks with clickbait raises CTR and lowers retention.
  • ML task: for one user at one request, score each candidate item (post, answer thread, community file, article) and return an ordered page of ~20 items. The model predicts several engagement probabilities per (user, item, context); a value function combines them into one ranking score.
  • Scale assumptions (state them out loud): 50M DAU, ~10 feed loads per user per day = 500M requests/day, about 5.8k QPS average and ~17k QPS at a 3x peak. Corpus of ~20M eligible items, ~2M new items/day. Most engagement goes to items under 48h old.
  • Latency: p99 under 200 ms server-side for the whole pipeline. That rules out scoring 20M items with a heavy model; we need a funnel: retrieve ~2,000, light-rank to ~500, heavy-rank 500, re-rank to 20.
  • ML objective vs business metric: offline we minimize per-task log loss and maximize ranking quality on logged impressions. Online we judge by sessions per user, engaged time and retention, with guardrails on hides, reports and latency. The value-function weights are the bridge between them, and they are tuned by A/B tests, not by offline loss.
  • Clarify early: surface (home feed only, not search), cold-start share of users, whether items are user-generated (moderation needed), and whether creators' reach is itself a goal (marketplace health).

Data and labels

  • Sources: impression logs (what was shown, position, request id, model version, features served), engagement events (click, dwell, like, save, share, comment, follow author, hide, report), item metadata and content, social graph (follows), user profile and settings.
  • Unit of training: one impression. Join engagements to impressions by request_id + item_id within an attribution window of 30 minutes.
  • Labels (one per head):
    • click: opened the item. Delay seconds.
    • long_dwell: dwell over 30 s after open (or over 10 s visible in-feed for items read inline). Delay up to the session end.
    • like, save, share, comment: explicit positives, sparse (0.1-1% of impressions).
    • hide / report / "see less": explicit negatives, rarer still, but heavily weighted in the value function.
    • Long-term labels (returned next day) are too delayed and too noisy per impression; use them for value-function tuning and as a slow auxiliary target, not as the main label.
  • Only count impressions that were actually seen: item at least 50% on screen for 1 s. Logged-but-unseen items below the fold are not negatives.
  • Imbalance: CTR ~3-5%, shares ~0.2%. Downsample negatives (keep rate r = 10%) for click-dense heads, keep all positives, then correct calibration: p = p' * r / (p' * r + 1 - p'). Do not downsample rare heads harder than their own positive rate supports.
  • Leakage traps:
    • Engagement counters (item CTR, author like count) must be computed point-in-time, as of the impression timestamp. A daily batch snapshot joined to the whole day leaks the label into the feature.
    • Best fix: log the exact feature values used at serving ("log and wait") and train on those. It removes train/serve skew too.
    • Do not use post-impression signals such as "user followed the author" as a feature for that same impression.
    • Same user's session split across train and test inflates offline numbers; split by time.
  • Position and selection bias: labels come from items the old model chose and placed. Log position, train with position as a feature, and set it to a fixed value (position 1) at serving. Keep 1-2% of traffic on a randomized-exploration policy to get less biased data and to evaluate counterfactually.
  • Splits: train on days D-30 to D-2, validate on D-1, test on D. Never random-split impressions. Also report on a cold-start slice (users under 7 days old, items under 2 h old).
  • Sampling for retrieval training: positives are long-dwell or explicit engagements; negatives are in-batch items with logQ correction for popularity, plus a small share of random corpus negatives.

Features

  • User (offline, daily batch): long-term topic/interest embedding, follow list size, account age, language, country, historical per-topic CTR, average session length, device mix.
  • User (online, streaming, seconds fresh): last 50 interactions (item ids, topics, action type, timestamp) as a sequence, items already shown in the last 24 h (for dedupe and fatigue), session depth, time since last visit.
  • Item (on publish): content embedding from a text/image encoder (title, body, thumbnail), topic classifier output, language, quality and safety scores, author id, age.
  • Item (streaming): impressions, clicks, likes, hides over 1 h / 24 h / 7 d windows; smoothed CTR (clicks + a) / (impressions + a + b) with a Beta prior so new items are not ranked on 1/1 = 100%.
  • Author: follower count, historical engagement rate, recent hide/report rate, posting frequency.
  • User x item cross: user-author affinity (past engagements with this author, decayed), user-topic CTR for the item's topic, cosine of user and item embeddings from the retrieval towers, follows author (yes/no), social proof (number of followed users who engaged).
  • Context: hour of day, day of week, device, app version, network type, feed position slot, which retrieval source produced the candidate.
  • Feature store: offline store (columnar, partitioned by hour) with point-in-time joins for training; online KV store for user and item features with p99 under 10 ms for a 500-key multiget. Streaming jobs (Flink-style) update counters and the recent-activity sequence within ~30 s. One feature definition generates both paths to keep them consistent.
  • Freshness matters most for: new-item counters and the user's in-session sequence. A user who just hid three crypto posts should not see a fourth on the next scroll.

Model

  • Baseline (ship first): candidates = recent items from followed authors + global trending in the user's top topics; rank with a logistic regression or GBDT on ~50 hand-made features predicting click. Score with recency decay. Gives a working feed, a logging pipeline, and a bar to beat.
  • Stage 1: candidate generation (~2,000 items, under 40 ms). Several sources run in parallel, each with a quota:
    • Two-tower model: user tower (profile + recent-sequence encoder) and item tower (content embedding + id embedding + metadata), 128-d output, trained with sampled softmax over in-batch negatives with logQ correction. Item embeddings in an ANN index (HNSW or ScaNN), top 800.
    • Follow graph: recent items from followed authors and topics, top 400 by recency and author affinity.
    • Item-to-item co-engagement from the user's last 20 engaged items, top 400.
    • Trending / fresh: high-velocity items under 6 h old in the user's languages, top 200.
    • Exploration: random high-quality new items, top 200, so new items gather impressions.
    • Merge, dedupe, drop seen and blocked items.
  • Stage 2: light ranker (~2,000 to 500, under 15 ms). Small two-tower or distilled MLP using cached embeddings and a few dozen cheap features; trained to mimic the heavy ranker's score (distillation) plus the click label.
  • Stage 3: heavy ranker (500 items, under 50 ms on GPU). Multi-task network: shared bottom of embeddings for sparse ids (user, item, author, topic), a target-attention block over the user's last 50 interactions (DIN-style) with the candidate as the query, a DCN-v2 cross layer, then MMoE with one tower per task (click, long_dwell, like, share, comment, hide). Loss: sum of per-task binary cross-entropy with task weights; long_dwell is conditional on click, so model p(dwell | click) and multiply. Position enters as a shallow side tower added to the logit and dropped at serving.
  • Value function: score = w1*p(click) + w2*p(click)*p(long_dwell|click) + w3*p(like) + w4*p(share) + w5*p(comment) - w6*p(hide), with w6 large (a hide costs about as much as several clicks are worth). Weights are tuned by online experiments against retention, not fit offline.
  • Stage 4: re-rank and policy (under 10 ms). Remove unsafe or low-quality items, cap at 2 items per author in the top 10, enforce topic diversity with MMR (score - lambda * max_sim_to_already_picked), boost fresh items slightly, interleave exploration slots (e.g. slot 5 and 15), apply business rules (no repeats in 24 h).
  • Cold start: new users get onboarding-topic embeddings and popularity within chosen topics; new items get their item-tower embedding from content alone at publish, so they are retrievable in minutes before any engagement exists.

Evaluation

  • Retrieval: recall@K of next-day long-dwell items per source and for the merged set (K = 2,000); coverage of items under 2 h old; source overlap.
  • Ranking, per head: log loss and normalized entropy (log loss divided by the entropy of the base rate), AUC and user-grouped AUC (AUC within each user, averaged, which matches how a feed actually ranks). Report on the cold-start slices separately.
  • Ranking, combined: NDCG@10 on the value-function score against a graded relevance (share > like > long dwell > click > none, hide negative).
  • Calibration: predicted over observed ratio per head, overall and by slice (new users, new items, country, device), target 0.95-1.05. It matters because the value function adds probabilities from different heads; a 20% over-predicted p(share) silently reweights the feed.
  • Offline limits: logged data only covers what the old model showed. Use the randomized-exploration traffic for unbiased replay (inverse propensity scoring) when comparing policies; treat offline gains as a gate, not a verdict.
  • Online A/B: user-level randomization, 1-5% per arm, at least 2 weeks to get past novelty effects.
    • Primary: sessions per user per day and daily engaged time; secondary: 7-day retention, long-dwell clicks per user.
    • Guardrails: hide and report rate, unfollows, p99 latency, crash rate, share of impressions to items under 24 h, author coverage (number of distinct authors receiving impressions), new-user retention.
    • Keep a long-term holdback (1% on the previous quarter's model) to catch slow effects like feed fatigue.
  • Interleaving for fast ranker comparisons: needs ~10x fewer users than an A/B for directional reads, then confirm with a full A/B.

Serving and monitoring

The request path, from candidates to the re-ranked feed:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":130,"width":180,"height":90,"label":{"text":"Client: feed\nrequest"}},
 {"type":"rectangle","id":"feed","x":260,"y":130,"width":180,"height":90,"label":{"text":"Feed service"}},
 {"type":"rectangle","id":"fs1","x":250,"y":-60,"width":200,"height":90,"label":{"text":"Online feature\nstore: user +\nsession"}},
 {"type":"rectangle","id":"cg","x":520,"y":130,"width":180,"height":90,"label":{"text":"Candidate\ngeneration"}},
 {"type":"rectangle","id":"ann","x":780,"y":-280,"width":180,"height":70,"label":{"text":"ANN index:\ntwo-tower"}},
 {"type":"rectangle","id":"follow","x":780,"y":-140,"width":180,"height":70,"label":{"text":"Follow graph\nsource"}},
 {"type":"rectangle","id":"i2i","x":780,"y":0,"width":180,"height":70,"label":{"text":"Item-to-item\nsource"}},
 {"type":"rectangle","id":"trend","x":780,"y":300,"width":180,"height":70,"label":{"text":"Trending +\nexplore source"}},
 {"type":"rectangle","id":"light","x":1040,"y":130,"width":180,"height":90,"label":{"text":"Light ranker:\n2000 to 500"}},
 {"type":"rectangle","id":"fs2","x":1030,"y":-80,"width":200,"height":90,"label":{"text":"Online feature\nstore: item +\ncross"}},
 {"type":"rectangle","id":"heavy","x":1300,"y":130,"width":180,"height":90,"label":{"text":"Heavy ranker\non GPU:\nmulti-task"}},
 {"type":"rectangle","id":"rerank","x":1560,"y":130,"width":200,"height":90,"label":{"text":"Re-rank:\ndiversity, policy,\nexplore slots"}},
 {"type":"arrow","id":"client-feed","x":180,"y":175,"width":80,"height":0,"start":{"id":"client"},"end":{"id":"feed"}},
 {"type":"arrow","id":"feed-fs1","x":350,"y":130,"width":1,"height":-100,"start":{"id":"feed"},"end":{"id":"fs1"}},
 {"type":"arrow","id":"feed-cg","x":440,"y":175,"width":80,"height":0,"start":{"id":"feed"},"end":{"id":"cg"}},
 {"type":"arrow","id":"cg-ann","x":570,"y":130,"width":210,"height":-375,"start":{"id":"cg"},"end":{"id":"ann"}},
 {"type":"arrow","id":"cg-follow","x":610,"y":130,"width":170,"height":-235,"start":{"id":"cg"},"end":{"id":"follow"}},
 {"type":"arrow","id":"cg-i2i","x":650,"y":130,"width":130,"height":-95,"start":{"id":"cg"},"end":{"id":"i2i"}},
 {"type":"arrow","id":"cg-trend","x":660,"y":220,"width":120,"height":115,"start":{"id":"cg"},"end":{"id":"trend"}},
 {"type":"arrow","id":"cg-light","x":700,"y":175,"width":340,"height":0,"start":{"id":"cg"},"end":{"id":"light"}},
 {"type":"arrow","id":"light-fs2","x":1130,"y":130,"width":1,"height":-120,"start":{"id":"light"},"end":{"id":"fs2"}},
 {"type":"arrow","id":"light-heavy","x":1220,"y":175,"width":80,"height":0,"start":{"id":"light"},"end":{"id":"heavy"}},
 {"type":"arrow","id":"heavy-rerank","x":1480,"y":175,"width":80,"height":0,"start":{"id":"heavy"},"end":{"id":"rerank"}},
 {"type":"arrow","id":"rerank-client","x":1660,"y":220,"width":-1570,"height":0,"points":[[0,0],[0,240],[-1570,240],[-1570,0]],"start":{"id":"rerank"},"end":{"id":"client"}}
]}

The loop that turns engagement into fresh features, labels and models:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"client","x":0,"y":0,"width":180,"height":90,"label":{"text":"Client"}},
 {"type":"rectangle","id":"ev","x":260,"y":0,"width":180,"height":90,"label":{"text":"Engagement\nevents"}},
 {"type":"rectangle","id":"stream","x":540,"y":0,"width":220,"height":90,"label":{"text":"Stream processing:\ncounters +\nsequences"}},
 {"type":"rectangle","id":"fs2","x":840,"y":0,"width":200,"height":90,"label":{"text":"Online feature\nstore: item +\ncross"}},
 {"type":"rectangle","id":"rerank","x":0,"y":240,"width":180,"height":90,"label":{"text":"Re-rank"}},
 {"type":"rectangle","id":"log","x":260,"y":240,"width":200,"height":90,"label":{"text":"Impression log\nwith served\nfeatures"}},
 {"type":"rectangle","id":"join","x":540,"y":240,"width":200,"height":90,"label":{"text":"Label join:\n30 min window"}},
 {"type":"rectangle","id":"train","x":820,"y":240,"width":200,"height":90,"label":{"text":"Training: ranker\nhourly, towers\ndaily"}},
 {"type":"rectangle","id":"reg","x":1100,"y":240,"width":200,"height":90,"label":{"text":"Model registry +\noffline gates"}},
 {"type":"rectangle","id":"heavy","x":1380,"y":100,"width":180,"height":90,"label":{"text":"Heavy ranker\non GPU"}},
 {"type":"rectangle","id":"idx","x":1380,"y":380,"width":200,"height":90,"label":{"text":"Item embedding\nindex rebuild"}},
 {"type":"rectangle","id":"ann","x":1660,"y":380,"width":180,"height":90,"label":{"text":"ANN index:\ntwo-tower"}},
 {"type":"arrow","id":"client-ev","x":180,"y":45,"width":80,"height":0,"start":{"id":"client"},"end":{"id":"ev"}},
 {"type":"arrow","id":"ev-stream","x":440,"y":45,"width":100,"height":0,"start":{"id":"ev"},"end":{"id":"stream"}},
 {"type":"arrow","id":"stream-fs2","x":760,"y":45,"width":80,"height":0,"start":{"id":"stream"},"end":{"id":"fs2"}},
 {"type":"arrow","id":"rerank-log","x":180,"y":285,"width":80,"height":0,"start":{"id":"rerank"},"end":{"id":"log"}},
 {"type":"arrow","id":"log-join","x":460,"y":285,"width":80,"height":0,"start":{"id":"log"},"end":{"id":"join"}},
 {"type":"arrow","id":"ev-join","x":390,"y":90,"width":210,"height":150,"start":{"id":"ev"},"end":{"id":"join"}},
 {"type":"arrow","id":"join-train","x":740,"y":285,"width":80,"height":0,"start":{"id":"join"},"end":{"id":"train"}},
 {"type":"arrow","id":"train-reg","x":1020,"y":285,"width":80,"height":0,"start":{"id":"train"},"end":{"id":"reg"}},
 {"type":"arrow","id":"reg-heavy","x":1300,"y":260,"width":80,"height":-115,"start":{"id":"reg"},"end":{"id":"heavy"}},
 {"type":"arrow","id":"reg-idx","x":1300,"y":310,"width":80,"height":115,"start":{"id":"reg"},"end":{"id":"idx"}},
 {"type":"arrow","id":"idx-ann","x":1580,"y":425,"width":80,"height":0,"start":{"id":"idx"},"end":{"id":"ann"}}
]}
  • Latency budget (p99, server-side, 200 ms): user features 10 ms, candidate generation 40 ms (sources in parallel), light ranker 15 ms, item/cross feature fetch 20 ms, heavy ranker 50 ms, re-rank 10 ms, serialization and network 25 ms; ~30 ms slack. If a source times out, serve without it rather than fail the request.
  • Capacity: 17k peak QPS x 500 items = 8.5M heavy-ranker inferences/s. Batch each request's 500 items in one GPU call; with ~100k items/s per GPU that is ~85 GPUs plus 30-50% headroom. Precompute and cache user-side tower outputs per session to cut cost.
  • Caching: cache the ranked page for 1-2 minutes so pull-to-refresh within a short window pages through the existing list; recompute when the session sequence changes materially.
  • Retraining cadence: heavy ranker warm-started and trained incrementally every 1-4 hours on fresh data (trends shift within hours); full retrain weekly. Towers retrained daily; new item embeddings computed at publish and inserted into the ANN index incrementally, full index rebuild daily. Every model passes offline gates (NE no worse than prod, calibration in range, slice checks) and a canary at 1% before ramp.
  • Monitoring:
    • Feature drift: PSI per feature daily against the training window; alert on null rate spikes (a broken streaming job looks like drift).
    • Prediction drift: mean predicted vs observed rate per head by hour; calibration alerts fire before A/B metrics move.
    • Data health: label join rate, impression-log volume, share of requests where a candidate source failed.
    • Business: hide rate, engaged time, author coverage dashboards by model version.
  • Feedback loops: the model trains on what it chose to show, so popular items get more popular and niche interests collapse. Counter with the exploration slots, per-author caps, diversity in re-ranking, and the position-debiasing tower. Watch the share of impressions going to the top 1% of authors over time.

Trade-offs

  • Multi-stage funnel vs one model: meets the 200 ms budget, but the heavy ranker never sees what retrieval dropped; retrieval recall caps total quality.
  • Multi-task value function vs single click model: captures dwell, shares and hides, but adds hand-set weights that need online tuning and can drift as heads recalibrate.
  • Hourly incremental training vs daily full retrain: fresher on trends, but higher risk of shipping a bad model from a broken data hour; needs automated gates and fast rollback.
  • Logging served features vs recomputing them: removes skew and leakage, but costs storage (hundreds of features x 500M requests/day) and slows adding a new feature (wait to accumulate logs, or backfill with care).
  • Negative downsampling: cuts click-head training data ~7x (95% negatives kept at 10%), but requires calibration correction and loses some hard negatives.
  • Exploration traffic: gives new items reach and unbiased data, but costs a little short-term engagement on those slots.
  • Diversity and author caps: protect long-term retention and creator health, but lower short-term CTR measured per impression.
  • GPU heavy ranker vs CPU GBDT: better quality with sequence features, but higher serving cost and a harder on-call story.

Interviewer expectations

Mid-level (L4)

  • Proposes a candidate-then-rank design with a sensible baseline.
  • Defines click and dwell labels from impression logs and splits by time.
  • Names AUC/log loss offline and an A/B test with a CTR or engagement metric.

Senior (L5)

  • Sizes QPS and builds a latency budget that justifies each funnel stage.
  • Multi-task ranker with a value function, explicit negatives (hide), and calibration because heads are combined.
  • Catches point-in-time leakage, position bias and the seen-impression definition; uses logged serving features.
  • Picks retention-oriented online metrics with guardrails; plans for cold-start users and items.

Staff (L6+)

  • Separates the business goal from the training objective and explains how value weights are tuned and governed.
  • Addresses feedback loops, exploration, creator-side health and long-term holdbacks.
  • Plans the platform: shared feature definitions, model gates, rollback, cost per 1k requests, and how multiple teams add candidate sources without breaking the feed.

Sign in to comment, vote or remix.