Design a notification system

A shared service other teams call to send push, email and SMS notifications: templates, user preferences, retries, deduplication and rate limits.

We are building a shared notification service that product teams call to send push, email and SMS, and its hardest caller: "someone you follow just posted", which fans one event out to every follower.

Functional requirements

• Producers submit an event (post.created, order.shipped, password.reset) or a direct send to one user; the service owns templates, channel choice and delivery.
• Templates are versioned per event type, channel and locale, rendered with event variables.
• Users set preferences per category and channel (mute "new posts" push, keep "security" email), plus quiet hours in their timezone.
• Follower fanout: one post event becomes a notification for each eligible follower, including authors with tens of millions of followers.
• Retries with backoff per channel; dedup so a producer retry or a worker crash never sends twice.
• Rate limits: per producer, per recipient (fatigue caps), and per provider (APNs, FCM, SES, Twilio quotas).
• An in-app inbox that clients read to recover anything push dropped.

Non-functional requirements

• Accepting an event is durable and fast: 202 within 50 ms p99, never lost after that.
• Transactional sends (OTP, password reset) reach the provider in under 5 s p99; social fanout may take minutes.
• At-least-once internally, effectively-once to the user via idempotency keys.
• A slow or down provider must not block other channels or other producers.
• Multi-region; a user's data and sends live in their home region.

Back-of-the-envelope estimation

• 500M users, 1.5B registered devices. 2B notification intents/day ≈ 23K/s average, 200K/s peak.
• Post events: 50M posts/day, average 200 followers → 10B candidate recipients/day before eligibility; ~80% are filtered out (muted, inactive, capped), leaving ~2B intents.
• A hot author with 50M followers posting once produces 50M candidates. At 1M follower reads/s that is 50 s of expansion work, so it must be chunked and paced.
• Intent row ≈ 300 B → 2B × 300 B ≈ 600 GB/day; keep 30 days hot ≈ 18 TB in the inbox store.
• Channel mix after aggregation: ~70% push, 25% in-app only, 4% email, 1% SMS (SMS ≈ 20M/day, the expensive one, ~$0.008 each).

Producer ingress (authenticated per producer service, idempotent):

POST /v1/events
{ "idempotency_key": "post:9f2c", "type": "post.created", "actor_id": "u_42",
  "audience": {"kind": "followers", "of": "u_42"}, "data": {"post_id": "p_9f2c", "snippet": "..."},
  "priority": "social" }
→ 202 { "event_id": "ev_..." }

POST /v1/sends   // direct, one recipient, transactional
{ "idempotency_key": "otp:u_7:1739", "user_id": "u_7", "template": "otp",
  "channels": ["sms"], "data": {"code": "481920"}, "priority": "critical", "ttl_s": 300 }
→ 202 { "intent_id": "ni_..." }

Device and preference surfaces (called by clients):

PUT    /v1/devices/{device_id}   { "platform": "ios", "token": "...", "app_version": "8.2", "locale": "en-GB", "tz": "Europe/London" }
DELETE /v1/devices/{device_id}
GET    /v1/preferences           → { "categories": {"new_posts": {"push": true, "email": false}}, "quiet_hours": {"start": "22:00", "end": "07:00"} }
PATCH  /v1/preferences           { "categories": {"new_posts": {"push": false}} }

Recovery read (the source of truth when push is lost):

GET /v1/inbox?cursor=...&limit=50 → { "items": [{ "intent_id", "type", "summary", "created_at", "read": false }], "next_cursor", "unread_count" }
POST /v1/inbox/read { "up_to": "ni_..." }

Internal operations: POST /admin/templates (new version, staged rollout), POST /admin/killswitch {category|provider|producer}, GET /v1/intents/{id} (delivery status trail for support).

Boundaries and invariants:

• The service is authoritative for intents, deliveries, devices and preferences. It does not own the follow graph or post content; it reads them.
• An intent is written once per (user, dedup_key). Delivery records hang off it and are append-only. Lifecycle: pending → aggregated → dispatched → (delivered | failed | expired | suppressed).
• The inbox is the durable promise; push, email and SMS are best-effort projections of it.
• Access paths: inbox reads by user_id newest-first; device lookup by user_id; token invalidation by token (hence the unique index).

flowchart LR
  P[Producers] --> API[Ingress API]
  API --> K1[(Event log)]
  K1 --> FO[Fanout planner]
  FO --> G[(Follow graph)]
  FO --> K2[(Recipient chunks)]
  K2 --> EL[Eligibility workers]
  EL --> PR[(Prefs and devices)]
  EL --> IN[(Inbox store)]
  EL --> K3[(Intent queue)]
  K3 --> AG[Aggregator]
  AG --> PQ[(Per channel queues)]
  PQ --> W[Provider workers]
  W --> APNS[APNs and FCM]
  W --> EM[Email and SMS providers]
  W -. invalid token .-> PR
  C[Client app] --> IN

Two layers: first record what the user should know (authoritative, durable), then try to tell them through channels (best effort, retried, droppable).

1. Accept. Ingress checks producer auth and quota, checks idempotency_key in Redis (SET NX, 24 h), appends to the event log and returns 202. Direct sends skip fanout and go straight to eligibility on a high-priority partition.
2. Expand. The fanout planner looks up the author's follower count. Under 10K followers it pages the follow graph itself and emits chunks of 1,000 recipient IDs. Above that it emits a job descriptor (author, cursor ranges) onto a separate hot-author lane, and a pool of workers expands ranges at a capped rate.
3. Eligibility. For each chunk, a worker batch-reads preferences and devices (one multi-get per 1,000 users), drops muted, blocked, inactive (no open in 90 days) and capped users, then writes an intent row per survivor to the inbox with dedup_key = post:{post_id}. Duplicate writes are no-ops because of the dedup check plus a deterministic intent_id = hash(user, dedup_key).
4. Aggregate. Intents for social categories go through a per-recipient aggregator that holds them for a short window and merges bursts ("Ana and 4 others posted"). Critical intents bypass it.
5. Deliver. Provider workers pull from per-channel, per-provider queues, render the template for the device locale, call the provider, and record the result. Invalid-token responses mark the device dead.
6. Recover. On open, the client calls GET /v1/inbox and shows the badge from unread_count, so a lost or suppressed push costs nothing but latency.

Hot-author fanout and fairness. A 50M-follower post must not starve the queue for everyone else, and must not hammer the follow graph.

Order the big expansions so the most engaged followers (recent opens, close ties) go first: the first 5% of the fanout gets most of the opens, and the long tail can wait minutes or be dropped to inbox only. Track per-author lag; if it exceeds 15 minutes, switch the remaining followers to inbox-only (no push), since a stale "X just posted" push is worse than none. Weighted fair queuing across producers keeps a marketing blast from delaying password resets: critical, transactional and social traffic use separate Kafka topics and separate worker pools.

Recipient burst coalescing. A user who follows 300 active accounts can get dozens of intents an hour. The aggregator keeps per-user state in Redis: agg:{user}:{category} holds pending intent IDs and a deadline. The first intent opens a 2-minute window (10 minutes for low-priority categories); later ones join it. At the deadline it emits one push: a single item if the window holds one, otherwise a summary with a collapse key so the device replaces rather than stacks. Fatigue cap: at most 6 social pushes per user per day and 1 per 30 minutes, kept as a sliding-window counter. Over the cap, the intent stays in the inbox with no push. Quiet hours and timezone-local batching use the same machinery: an intent created at 02:00 local is parked until 07:00 local and merged into a morning digest, with jitter over 30 minutes so a whole timezone does not hit APNs at once.

Retries, idempotency and provider limits. Each delivery attempt carries delivery_id = hash(intent_id, channel, device_id), sent as the provider's idempotency or collapse ID where it has one (APNs apns-collapse-id, email Message-ID). Retry policy depends on the error:

Each provider has a token bucket sized to its quota (for example SES 500/s per account, Twilio per-number throughput), shared across workers through Redis with a local allowance per worker. A circuit breaker opens when the error rate passes 20% over 30 s; while open, push falls back to inbox only and critical SMS fails over to a second SMS vendor. Kill switches per category, producer or provider are flags read every few seconds, for a runaway template or a buggy producer.

Token lifecycle. Tokens die quietly: uninstalls, reinstalls, OS restores. Rules: on app start the client always re-registers its token (upsert keyed by device_id, unique on token so a token that moved to a new install detaches from the old device); provider invalid responses set invalid_at and stop sends at once; devices unseen for 270 days are pruned; on logout the device is deleted so a shared phone does not get the previous user's DMs. Feedback-driven cleanup keeps invalid-token rates under 1%, which matters because providers throttle senders with high failure rates.

Regional routing. Each user has a home region that holds their inbox, devices and preferences. A post in region A whose followers live in B and C is fanned out by the planner in A into per-region chunk streams; each region's eligibility workers write to their local stores. Only the event and recipient ID lists cross regions, never device tokens, which also helps with data residency.

• Writing the inbox before sending gives a recoverable source of truth at the cost of ~2B writes a day; sending straight from the queue is cheaper but loses anything a provider drops.
• Aggregation windows reduce noise and provider cost, but add up to 2–10 minutes of latency to social pushes; critical traffic opts out.
• Engagement-ordered hot-author fanout delivers value fast, but the last followers can be minutes late or downgraded to inbox only.
• At-least-once plus dedup keys is simpler and more robust than exactly-once machinery, but needs a dedup store with TTL and still allows a rare double send if a provider times out after accepting.
• Reading preferences at eligibility time (not at send) means a mute takes effect for new intents only; already-queued pushes in the next couple of minutes still go out unless the worker rechecks a cached flag.
• Per-region homing keeps tokens local and latency low but makes cross-region fanout two hops and complicates users who move.

Mid-level (L4)

• Queue between ingress and senders, per-channel workers, templates and a preferences check.
• Retries with backoff and an idempotency key on the API.
• Reasonable tables for devices, preferences and notification history.

Senior (L5)

• Separates durable intent (inbox) from best-effort delivery and uses the inbox for client recovery.
• Handles hot-author fanout with chunking and a separate lane; isolates critical from bulk traffic.
• Token invalidation from provider feedback, provider rate limits, fatigue caps and quiet hours.

Staff (L6+)

• Explicit lifecycle and invariants (one intent per user and dedup key, append-only deliveries) and who owns which data.
• Fairness and degradation policy: engagement ordering, lag-triggered downgrade to inbox only, circuit breakers, kill switches, SMS failover.
• Regional homing, cost of SMS and email, and how to measure success (open rate, opt-out rate, end-to-end lag per priority).