Design a content moderation system

Source: https://www.hack2hire.com/question-bank/questions/system-design/69e841e1fbaf3da641828244
Company: Meta
Difficulty: Hard

Problem statement

Screen user posts, images and videos for policy violations with ML classifiers and human reviewers, acting within seconds on the worst content.

Requirements

We are building the moderation pipeline for a large social network: every post, comment, image and video is checked against policy by ML classifiers, uncertain cases go to human reviewers, and the worst content (CSAM, terrorism, credible threats) is blocked or removed within seconds.

Functional requirements

• Screen text synchronously before it publishes; screen images and video asynchronously, with a fast first pass and a deeper pass after.
• Each item gets one canonical decision per policy: allow, limit (no recommendations, interstitial, age gate), remove, or escalate, with the policy, model version and reason that produced it.
• Route uncertain or high-reach items to human review queues by policy, language and severity.
• Users can report content and appeal a decision; appeals go to a different reviewer.
• Roll up item decisions into account strikes, feature limits and bans.
• Policies vary by region and surface (a teen DM is stricter than a public adult post).

Non-functional requirements

• Text gate adds under 150 ms p99 to the post path; it must never block posting for more than that.
• Known-bad media (hash match) is removed in under 5 s; severe novel content in under 60 s; everything else classified within 10 min.
• Decisions are durable and auditable for years (legal holds, regulator requests, DSA transparency reports).
• Degrade safely: if classifiers are down, publish low-risk content and hold high-risk content, never silently skip moderation.
• Reprocessing a policy or model change over billions of items must not starve live traffic.

Back-of-the-envelope estimation

• 1B posts/day text ≈ 12k/s average, 40k/s peak. 300M images/day ≈ 3.5k/s; 30M videos/day ≈ 350/s.
• Text: 5 classifiers × 40k/s = 200k inferences/s at ~5 ms each on GPU batches → a few hundred GPUs.
• Images: perceptual hash + 3 vision models; video: sample 1 frame/s plus audio ASR. Average video 60 s → 60 frames × 350/s ≈ 21k frame inferences/s.
• About 0.5% of items land in human review → 6M/day. At 60 s average handling and 80% occupancy, that is ~2,100 reviewer-hours per hour, roughly 15k reviewers across shifts and languages. Every 0.1 pt of auto-decision precision is worth thousands of reviewers.
• Decision records: 1.3B items × ~5 policy rows × 200 B ≈ 1.3 TB/day before compression.

API design

Producers (post service, upload service, comment service) call a gRPC gate; everything else is event-driven over Kafka.

• Moderate(ModerateRequest) → ModerateResponse (sync, text and metadata)
  • Request: {item_id, item_type: "post", author_id, surface: "feed", region: "DE", text, media_ids: [...], idempotency_key}
  • Response: {decision: "allow" | "hold" | "remove", visibility: "public" | "author_only", decision_id, policy_set_version, pending_media: true}
  • hold means "publish to author only until async checks finish".
• Event media.uploaded {item_id, media_id, blob_uri, kind, author_id, surface} → async pipeline; result event moderation.decided {item_id, decision_id, action, policies[], source: "model" | "human" | "hash"} that the content service consumes to change visibility.
• GET /v1/items/{item_id}/decision → current canonical decision with history (for support tools, the content service on cache miss, and recovery after a lost event).
• POST /v1/reports {item_id, reporter_id, reason} → 202 {report_id}; reports aggregate into a review signal, they don't each create a task.
• POST /v1/appeals {decision_id, user_id, text} → 202 {appeal_id, status: "queued", sla_hours: 24}; one active appeal per decision.
• Reviewer surface: POST /v1/review/next {reviewer_id} → leased task; POST /v1/review/tasks/{id}/verdict {action, policy, notes}.

Why this shape: the post path needs an answer now, so text is sync with a hard deadline; media inference takes seconds to minutes, so it is async and pushes the outcome back as an event. The read endpoint exists because events can be lost or replayed out of order and a client needs one source of truth.

Data model

Invariants:

• Decisions are never updated, only appended; the item's current decision is the latest row whose precedence wins (human > hash > model; appeal outcome > original).
• Every decision records the model_version and policy_version that produced it, so any decision can be reproduced and any version rolled back.
• A review task has at most one active lease; verdicts carry the lease token so a stale reviewer can't overwrite.

Wide-column fits the write-heavy, item-keyed decision log; the review queue needs transactional leasing and secondary ordering, so it lives in a relational store.

High-level design

flowchart LR
  P[Post service] -->|sync text| G[Moderation gate]
  G --> TC[Text classifiers]
  G --> HB[Hash bank]
  U[Upload service] --> K[(Kafka)]
  G --> K
  K --> MP[Media pipeline]
  MP --> HB
  MP --> VC[Vision and audio models]
  MP --> DE[Decision engine]
  G --> DE
  DE --> DS[(Decision store)]
  DE -->|uncertain| RQ[Review queues]
  RQ --> RT[Reviewer tool]
  RT --> DE
  R[Reports and appeals] --> RQ
  DE --> EN[Enforcement and rollups]
  DE -->|moderation decided| CS[Content service]
  PS[Policy config] -.-> DE

1. Text, pre-publish. The post service calls the gate with a 150 ms budget. The gate fans out to text classifiers (hate, harassment, spam, self-harm, violent threat) and a URL/phrase blocklist in parallel, then the decision engine applies the policy set for (region, surface). Above the remove threshold: reject with a reason. Below the allow threshold: publish. In between: publish with limited distribution and enqueue for review.
2. Media, async. Uploads land in blob storage and emit media.uploaded. Stage 1 (seconds): perceptual hash lookup against the hash bank plus a cheap image classifier; a hash hit on CSAM/terror removes immediately and files the legal report. Until stage 1 completes, the post is visible only to its author. Stage 2 (minutes): heavier vision models, video frame sampling, ASR + text classifiers on the transcript, OCR on images. Each stage writes a decision and emits moderation.decided.
3. Human review. Items with scores in the uncertainty band, high reach (going viral), or enough user reports become review tasks. Reviewers lease a task, see the item with the policy and model highlights, and submit a verdict, which becomes a human decision that overrides the model.
4. Appeals. An appeal creates a task in an appeal queue, routed to a reviewer who did not make the original call. Overturns restore the content and reverse any strike.
5. Enforcement. Every remove decision feeds the account rollup, which updates strikes and restrictions.
6. Feedback. Human verdicts and appeal outcomes become labeled training data and precision/recall metrics per model version.

Deep dives

Sync gate vs async deep moderation, and degraded modes. Only cheap checks go on the post path: text models distilled to small transformers (~5 ms on GPU with batching) and exact/near hash lookups. Everything expensive runs async with "author-only" visibility until stage 1 finishes, so the user sees their post instantly and nobody else sees unscreened media. The gate has a hard deadline; a classifier that misses it is treated as "no score". Degraded modes by risk:

Fail-open is never silent: every skipped check writes a needs_rescan marker and a backfill job drains it.

Human-in-the-loop queue: priority, fairness, surges. Queues are keyed by (policy, language, severity) so tasks only go to reviewers with the right training and language. Priority = severity_weight × P(violation) × expected_views_next_hour, so a borderline post going viral jumps ahead of a borderline post with 10 views; expected views come from current velocity. Tasks are leased for 5 min with heartbeat; an expired lease returns the task. Fairness: cap the share of any one queue a single high-volume reporter or brigading campaign can fill (dedupe reports per item, weight reporters by past accuracy), and age-boost priority so low-severity items still finish within SLA. Surge handling (breaking news, coordinated attack): raise the auto-action threshold temporarily for the affected policy, cluster near-duplicate items (same hash or embedding) so one verdict applies to thousands, and spill to vendor queues. Quality: 2–5% of tasks are double-reviewed or golden-set items to measure reviewer accuracy.

Policy and model versioning, safe reprocessing. A policy set is an immutable, versioned bundle: which models run, thresholds per (policy, region, surface), and action mapping. Rollout of a new model or threshold goes through: offline eval on the labeled set, shadow mode (score live traffic, log decisions, take no action), then a 1% → 10% → 100% ramp with dashboards comparing action rates against shadow. Reprocessing after a policy change (say, a new medical-misinformation rule) runs from the item store through a separate, rate-limited Kafka topic and GPU pool with lower priority than live traffic, newest and most-viewed items first. Reprocessed decisions are written with the new versions; a guard prevents reprocessing from overturning a human decision unless the policy itself changed. A bad rollout is reverted by switching the active policy version, and a replay job reverses actions taken by the bad version.

Account-level enforcement. Item decisions roll up into a per-account record: strikes per policy with 90-day expiry, and a ladder (warning → 24 h posting limit → 30-day restriction → disable). Severe single violations (CSAM, terror) skip the ladder. Rollups recompute when an appeal overturns a strike, so the ladder state is derived from the decision log, not a counter that drifts. Account signals (age, past strikes, network of banned accounts) also feed back into item thresholds.

Regional variants, drift and reviewer wellbeing. Region and surface select the threshold column in the policy set; illegal-in-country content is geo-restricted (limit in that region) rather than removed globally. Drift monitoring tracks per model version: action rate, review overturn rate, appeal overturn rate, and score distribution, alerting when any moves more than a set band day over day. Reviewers of graphic queues get blurred and grayscale previews by default, audio off, capped daily exposure, rotation off severe queues, and mandatory breaks.

Trade-offs

• Sync text gating catches violations before anyone sees them but adds latency and couples posting to model availability; async-only is faster to post but leaves a window where harmful content is live.
• Author-only visibility for unscanned media protects viewers but delays reach for every creator by seconds.
• Low auto-remove thresholds catch more harm but increase false positives, appeals and user anger; high thresholds push volume to reviewers, who are the most expensive and slowest part of the system.
• Fail-open on low-risk surfaces keeps the product up during outages but accepts a rescan backlog and some exposure; fail-closed on high-risk surfaces gives up availability there.
• Append-only decisions give a complete audit trail at the cost of storage and a read-time "latest wins" resolution.
• Clustering near-duplicates scales reviewers 100× during surges but one wrong verdict applies to the whole cluster.
• Geo-restricting instead of global removal respects local law but needs per-region policy sets that are harder to test.

Interviewer expectations

Mid-level (L4)

• Splits fast sync text checks from async media checks and explains why.
• Has a classifier score → threshold → action flow with a human review queue for the uncertain band.
• Stores decisions with enough context to support appeals.

Senior (L5)

• Sizes the human review load and shows why precision drives reviewer headcount.
• Prioritizes the review queue by severity and expected reach, with leasing and SLAs.
• Defines degraded modes per risk tier and a rescan path so fail-open is never silent.
• Versions models and policies, uses shadow mode and staged rollout, and keeps decisions append-only.

Staff (L6+)

• Treats policy as versioned configuration by region and surface, with safe bulk reprocessing that doesn't starve live traffic or overturn human decisions.
• Designs surge handling (near-duplicate clustering, threshold shifts) and adversarial resistance (brigaded reports, hash evasion).
• Ties in account enforcement derived from the decision log, drift monitoring, legal reporting obligations and reviewer wellbeing as first-class requirements.