Design a webhook delivery system

Source: https://www.hack2hire.com/question-bank/questions/system-design/69cb3b1aff3ebde766cedc63
Company: Stripe, OpenAI
Difficulty: Medium

Problem statement

Deliver event notifications to customers' HTTP endpoints at least once, with retries and backoff, signing, ordering where it matters, and isolation from slow endpoints.

Requirements

We are building the outbound webhook platform for an API company (think payment or model-API events): internal services publish events, and the platform delivers each one to every customer HTTPS endpoint subscribed to it, at least once, signed, retried with backoff, and without one broken endpoint slowing anyone else.

Clarifying questions worth asking first: Is ordering required globally, per object, or not at all? How long do we keep retrying (hours or days)? Do customers need replay and a delivery log in the dashboard? Are payloads full objects ("thin" vs "fat" events)? What fan-out per event is typical?

Functional requirements

• Customers register endpoints (URL, subscribed event types, optional ordering mode) and get a signing secret.
• Internal producers publish events; each event is delivered to every matching enabled endpoint.
• Failed deliveries retry with exponential backoff and jitter for up to 3 days, then land in a dead-letter state.
• Every request is signed (HMAC over timestamp + body) so receivers can verify origin and reject replays.
• Optional per-endpoint ordering: events sharing an ordering key (e.g. customer_id) arrive in order.
• Customers see the attempt log, can resend a single event, and can bulk-replay dead letters.

Non-functional requirements

• Durable accept: once the publish call returns 202, the event is never lost (RPO 0 for accepted events).
• At-least-once delivery; duplicates allowed and documented, receivers dedupe on event_id.
• p99 under 5 s from publish to first attempt for healthy endpoints.
• Isolation: an endpoint that times out for hours must not delay delivery to other customers.
• Egress safety: never let a customer URL reach internal networks (SSRF).
• Availability 99.99% on the publish path; delivery may lag during incidents but must catch up.

Back-of-the-envelope estimation

• 500M events/day ≈ 6k/s average, 30k/s peak. Average fan-out 2 endpoints → 12k deliveries/s average, 60k/s peak.
• Payload ~2 KB → 1 TB/day of event bodies; keep 30 days for replay → ~30 TB (object storage or a compressed column store, not the hot DB).
• Attempt log rows ~200 B; with ~1.3 attempts per delivery → ~16k rows/s, ~280 GB/day; keep 30 days hot.
• In-flight requests = rate × latency. Healthy endpoints at 300 ms: 60k × 0.3 = 18k concurrent. If 2% of traffic hits endpoints that hang to the 10 s timeout, that 2% alone needs 1.2k × 10 = 12k slots. This is why slow endpoints need their own capacity budget.
• 1M registered endpoints, ~50k receiving traffic at any minute.

API design

Customer-facing management API:

• POST /v1/webhook_endpoints {url, enabled_events: ["invoice.paid", ...], ordering: "none"|"per_key", description} → {id: "we_123", secret: "whsec_...", status: "enabled"}. The URL is validated (HTTPS, public IP, resolves) before saving.
• PATCH /v1/webhook_endpoints/{id} {url?, enabled_events?, status?}; POST /v1/webhook_endpoints/{id}/rotate_secret {expire_old_in: "24h"} → new secret, old one keeps signing until expiry.
• GET /v1/webhook_endpoints/{id}/deliveries?status=failed&cursor=... → attempt log with status codes, latency, truncated response body.
• POST /v1/deliveries/{id}/retry (single resend) and POST /v1/webhook_endpoints/{id}/replay {status: "dead", from, to} → {job_id} (bulk, rate-limited).

Internal publish API (producers):

• POST /internal/events {idempotency_key, account_id, type, ordering_key?, data} → 202 {event_id}. Same idempotency_key within 24 h returns the original event_id. Returns 202 only after the event is committed; a 5xx means the producer must retry with the same key. Producers in the same database as the business change use a transactional outbox instead, so the event exists iff the business write committed.

Outbound contract documented for receivers:

• POST <url> with JSON {id: "evt_...", type, created, data}, headers Webhook-Id, Webhook-Timestamp, Webhook-Signature: t=1760000000,v1=<hex hmac>[,v1=<hmac with old secret>].
• Any 2xx within 10 s is success. 3xx is not followed (counted as failure). 410 Gone disables the endpoint. Everything else, timeouts and TLS errors are retried.
• Duplicates and (without ordering mode) reordering are possible; dedupe on id, reject timestamps older than 5 minutes, and acknowledge fast then process asynchronously.

Data model

Endpoint config and delivery state live in a sharded relational store (Postgres/Vitess, sharded by account_id): it needs transactions for the "accept + enqueue" step and secondary indexes for the dashboard. Event bodies and old attempts go to object storage / a columnar store.

Invariants: one delivery row per (event, endpoint), enforced by the unique key, so fan-out can be retried safely. Status only moves forward (pending → in_flight → succeeded|retrying → ... → dead), and a transition is a conditional update on the current status and attempt_count, so a slow worker whose lease expired cannot overwrite a newer result. For ordered endpoints, seq is assigned per (endpoint_id, ordering_key) at fan-out.

High-level design

flowchart LR
  P[Producer services] --> API[Publish API]
  API --> DB[(Events and deliveries DB)]
  DB --> OB[Outbox relay]
  OB --> K[(Kafka events topic)]
  K --> F[Fan out workers]
  F --> DB
  F --> Q[(Per endpoint queues)]
  RS[Retry scheduler] --> Q
  DB --> RS
  Q --> D[Dispatch workers]
  D --> EG[Egress proxy pool]
  EG --> R[Customer endpoints]
  D --> DB
  D --> CB[Endpoint health and breaker]
  CB --> D

1. Durable accept. The publish API dedupes on idempotency_key, writes the event row and an outbox row in one transaction, stores the body, and returns 202. The outbox relay tails committed rows into Kafka (partitioned by account_id).
2. Fan-out. Fan-out workers load the account's endpoint subscriptions from a cache, insert one deliveries row per matching endpoint (idempotent on the unique key), and push a small work item {delivery_id, endpoint_id} onto that endpoint's queue.
3. Attempt and classify. A dispatch worker leases items per endpoint, signs the body, sends it through the egress proxy with a 10 s timeout, and classifies: 2xx → succeeded; 410 → disable endpoint; 4xx/5xx/timeout/TLS → retryable; DNS points to a private IP → permanent failure. It writes the attempt row and updates the delivery conditionally.
4. Retry. A retryable failure sets status=retrying, next_attempt_at=now+backoff(n). The retry scheduler polls due rows per shard and re-enqueues them. The endpoint's health record is updated, which can open its circuit breaker.
5. Dead letter. After 16 attempts (~3 days) the delivery becomes dead, the customer is emailed, and the row stays replayable for 30 days. An endpoint failing every attempt for 3 days is auto-disabled.

Not in scope here: inbound webhooks, event schema versioning, and the dashboard UI.

Deep dives

Retry schedule and dead-letter lifecycle. Backoff min(base × 2^n, 6h) with full jitter, base 30 s: attempts at roughly 30 s, 1 m, 2 m, 4 m ... then every 6 h, 16 attempts over ~72 h. Retries are rows with next_attempt_at, not delayed broker messages, so the schedule survives restarts and is queryable for the dashboard. The scheduler on each shard runs SELECT ... WHERE status='retrying' AND next_attempt_at <= now() ORDER BY next_attempt_at LIMIT 1000 FOR UPDATE SKIP LOCKED, marks them pending, and enqueues. In-flight leases expire after 30 s; a sweeper returns expired in_flight rows to retrying, which is one source of duplicates (the receiver may have processed the request we never heard back from). Dead letters are rows in dead status, not a separate queue, so replay is "set status pending, attempt_count stays, enqueue" with the current URL and secret.

Isolation from slow endpoints. The failure mode to prevent is shared worker threads all stuck on 10 s timeouts to one dead endpoint. Mechanisms: (1) per-endpoint queues with a concurrency cap (default 10 in-flight per endpoint, raised for big customers), so one endpoint can use at most its cap no matter how much backlog it has; (2) dispatch uses non-blocking I/O, so a hanging connection costs a socket and a timer, not a thread; (3) a per-endpoint circuit breaker: after 5 consecutive failures, open for 1 minute, then send one probe; while open, items are rescheduled without an HTTP call; (4) separate worker pools by health class, "healthy" and "degraded", and an endpoint that has been failing for over 10 minutes moves to the degraded pool with a small fixed budget. A customer's endpoint returning 500 for hours therefore costs one probe a minute plus its retry schedule, and its backlog drains at its cap when it recovers, rather than flooding it.

Ordering where it matters. Most endpoints get no ordering: parallel delivery, events carry created and the receiver re-fetches the object if needed. For ordering=per_key endpoints, the queue is partitioned by (endpoint_id, ordering_key) and only the lowest unacked seq per key is eligible. A failing event blocks later events for that key only (head-of-line blocking is the price). After the event goes dead, the key either stays blocked until the customer replays or skips it (customer chooses; default skip and flag). Global ordering per endpoint is refused: one bad event would stall everything.

At-least-once and the receiver contract. Duplicates come from lease expiry after a slow 2xx, worker crashes between HTTP response and DB write, manual resends, and replay. We can't remove them, because the ack is the HTTP response over an unreliable network. So we make them cheap for the receiver: stable id across all attempts of the same event to the same endpoint, docs telling them to store processed ids for 7 days with a unique constraint, and to return 2xx quickly and process in a queue. Exactly-once is not offered.

Signatures, secrets and egress safety. Signature is HMAC-SHA256(secret, "{timestamp}.{body}") over the raw bytes; receivers compare in constant time and reject timestamps more than 5 minutes off, which blocks replay of captured requests. Rotation: the endpoint holds up to two secrets; during the overlap window every request carries two v1= signatures, so the receiver can deploy the new secret at any time. Retries in flight are signed at attempt time with whatever secrets are current, never with the secret from the first attempt. URL edits likewise apply to pending retries, since the delivery stores endpoint_id, not the URL. Secrets are encrypted with a KMS key and decrypted in the dispatcher's memory cache. Egress: all requests go through a dedicated proxy pool in an isolated network with fixed published IPs (customers allowlist them). The proxy resolves DNS itself and refuses private, loopback, link-local and metadata ranges (10/8, 172.16/12, 192.168/16, 127/8, 169.254/16, IPv6 equivalents) at connect time, which defeats DNS rebinding; redirects are not followed; response bodies are capped at 64 KB.

Draining a regional backlog. After a broker or worker outage, millions of deliveries are due at once. Dispatchers keep per-endpoint caps, so recovery can't stampede any single customer; the global rate is raised gradually (e.g. +20% per minute) while watching egress proxy saturation; new events and retries share the queue but new events go first within an endpoint's budget, so fresh traffic recovers before the old backlog. Retries whose next_attempt_at was missed during the outage do not count the outage as attempts.

Trade-offs

• At-least-once with receiver dedupe gives up exactly-once semantics; in exchange the sender stays simple and never loses an event.
• DB-backed retry state costs index scans and write load (~16k row updates/s) versus a broker delay queue, but gives exact schedules, dashboards and replay from one source of truth.
• Per-endpoint queues and caps isolate customers but add millions of logical queues and a scheduler to pick fairly among them; a shared FIFO is simpler and fails badly under one hung endpoint.
• Per-key ordering is opt-in because it brings head-of-line blocking and lower throughput; unordered delivery scales but pushes reconciliation onto receivers.
• Fat payloads save receivers a fetch but leak more data if an endpoint is misconfigured and go stale on retry; thin events plus a fetch API are safer but double load.
• Auto-disabling endpoints after 3 days of failure protects capacity but can surprise a customer; email warnings at 1 h and 24 h soften it.

Interviewer expectations

Mid-level (L4)

• Durable accept then async delivery via a queue; workers POST and retry with exponential backoff.
• HMAC signing and a DLQ after N attempts.
• Knows delivery is at-least-once and that receivers must dedupe on event id.

Senior (L5)

• Transactional outbox or equivalent so accepted events are never lost; idempotent fan-out keyed on (event, endpoint).
• Per-endpoint concurrency caps and circuit breakers; sizes in-flight capacity with rate × latency.
• Timestamped signatures, dual-secret rotation, and SSRF-safe egress through a proxy with fixed IPs.
• Explains which failures produce duplicates and how lease expiry and conditional updates keep state correct.

Staff (L6+)

• Chooses ordering scope deliberately (opt-in per key) and defines behavior when an ordered event goes dead.
• Designs replay, backlog draining and gradual ramp-up after incidents; fairness between new events and retries.
• Treats the receiver contract as a product: documented semantics, dashboards, auto-disable policy and customer communication.
• Discusses cost: tiered storage for bodies and attempts, retention, and degraded-pool budgets.