Design a CI/CD system

Source: https://www.hack2hire.com/question-bank/questions/system-design/69c9ff24ff3ebde766ced6fa
Company: OpenAI
Difficulty: Hard

Problem statement

On every commit, build and test the code on a fleet of workers, then roll it out to production safely, with logs, retries and rollbacks.

Requirements

We are building a multi-tenant CI/CD service, in the spirit of GitHub Actions or Buildkite hosted runners: many organizations connect their Git repos, every push triggers a workflow defined in the repo, a shared fleet of ephemeral workers builds and tests it, and passing builds roll out to production in stages with logs, retries and one-click rollback.

Clarify early: are steps a linear list or a DAG (assume ordered stages, each stage a list of steps run in order; parallel jobs are an extension)? Do we host the runners or do tenants bring their own (we host, on Kubernetes)? Is "deploy" part of the pipeline or a separate service (a deploy is just a privileged step type that talks to a rollout controller)?

Functional requirements

• Ingest push and pull-request webhooks from the Git host; create at most one run per (repo, commit, workflow, trigger).
• Read the workflow file (.ci/pipeline.yml) at that exact commit and freeze it as the run's definition.
• Run steps in order on isolated, ephemeral workers; each step has a timeout, retry policy, env, secrets and cache keys.
• Stream logs live to the browser and keep them searchable for 90 days; store artifacts (test reports, built images).
• Cancel a run, retry a failed step, re-run a whole workflow; a new push to the same branch can auto-cancel the older run.
• Deploy steps: canary → 10% → 50% → 100% with health gates, automatic rollback on bad metrics, manual rollback to any prior release.

Non-functional requirements

• No lost or duplicated work: a webhook delivered three times makes one run; a step is never "running" on two workers and never silently forgotten.
• Push-to-first-step-start under 10 s p50, 30 s p95 when capacity exists.
• Tenant isolation: one tenant's code can't read another's secrets, source, caches or network.
• Fairness: a tenant that pushes 5,000 jobs can't starve everyone else.
• Control plane 99.95%; a runner node dying costs at most that step's attempt, not the run.
• Logs and live status degrade independently: if streaming is down, builds still run.

Back-of-the-envelope estimation

• 50k tenants, 500k repos; 5M runs/day ≈ 60 runs/s average, 300/s at the weekday peak.
• 10 steps per run → 50M step attempts/day, ~600/s average dispatches at peak, 1,500/s with burst.
• Mean step 3 min → ~600/s × 180 s ≈ 100k concurrent step pods at peak. At 4 pods per 16-vCPU node that is 25k nodes; autoscaling matters more than anything else for cost.
• Logs: 2 MB per step → 100 TB/day raw, ~20 TB compressed, 90-day retention ≈ 1.8 PB in object storage. Never in the database.
• Control-plane rows: 50M attempts × ~500 B ≈ 25 GB/day; events ~10× that. Partition by time, keep hot 30 days.

API design

Webhook ingest (Git host → us), must answer within 10 s or the host retries:

POST /hooks/{provider}
Headers: X-Signature-256, X-Delivery-Id
200 {"accepted": true}        # after a durable write, before any planning

Internal planning handoff: the ingest service writes the delivery and enqueues plan_push{delivery_id}; planners consume it.

Public REST (UI, CLI, other tools):

GET  /v1/repos/{repo_id}/runs?branch=main&status=failed&cursor=...
GET  /v1/runs/{run_id}
  → {run: {id, repo_id, commit_sha, ref, status, version, created_at,
           definition_digest},
     steps: [{id, ordinal, name, status, attempt_no, started_at,
              finished_at, exit_code, log_url}]}
POST /v1/runs/{run_id}/cancel
POST /v1/runs/{run_id}/steps/{step_id}/retry
POST /v1/repos/{repo_id}/runs  {"ref": "main", "workflow": "deploy"}   # manual trigger
GET  /v1/steps/{step_id}/attempts/{n}/logs?offset=0                   # paged
POST /v1/deployments/{deploy_id}/rollback  {"to_release": "rel_812"}

GET /v1/runs/{id} is the canonical snapshot for the run page and carries a version (the run's event sequence number).

Live progress: GET /v1/runs/{run_id}/events?after_seq=1834 as Server-Sent Events. Each event has a seq; on reconnect the client sends Last-Event-ID, and if that seq is older than the retained buffer the server replies reset and the client re-fetches the snapshot.

Runner protocol (worker agent → control plane, gRPC): Heartbeat(attempt_id, lease_token), AppendLog(attempt_id, chunk_seq, bytes), Complete(attempt_id, lease_token, exit_code, artifacts[]).

Data model

Control plane in a sharded relational store (Postgres via Citus, or Spanner/Vitess): runs and steps need transactions for "finish one step, release the next" and the invariants below. Shard key tenant_id, so a run and all its steps live on one shard and the scheduler's transaction stays local.

Indexes: runs (repo_id, ref, created_at desc) for the branch list; steps (status, ready_at) filtered to status='queued' for dispatch; attempts (lease_expires_at) filtered to running for the reaper.

Large payloads (raw webhook bodies, logs, artifacts, the junit XML) go to object storage keyed logs/{tenant}/{run}/{step}/{attempt}/{chunk_seq}; rows hold only references.

Invariants worth enforcing in the schema, not just code: one run per trigger (unique key); at most one non-terminal attempt per step (partial unique index on attempts(step_id) where status in ('leased','running')); a step at ordinal k is runnable only if step k−1 succeeded; status moves only forward (queued → leased → running → succeeded|failed|cancelled), enforced with conditional updates.

High-level design

flowchart LR
  GH[Git host] -->|webhook| IN[Ingest]
  IN --> DLV[(Deliveries)]
  IN --> Q1[Plan queue]
  Q1 --> PL[Planner]
  PL -->|read config at sha| GH
  PL --> DB[(Control plane DB)]
  DB --> SCH[Scheduler]
  SCH --> DQ["Dispatch queues per tenant"]
  DQ --> RC[Runner controller]
  RC --> K8S["Kubernetes runner pools"]
  K8S -->|heartbeat logs complete| RA[Runner API]
  RA --> DB
  RA --> LOG[(Log store)]
  DB --> EV[Event bus]
  EV --> SSE[Live gateway]
  SSE --> UI[Browser]
  SCH --> RO[Rollout controller]
  RO --> PROD[Production clusters]

1. Push → run. Ingest verifies the HMAC, inserts (provider, delivery_id) with insert-if-absent, stores the body in object storage, enqueues plan_push and returns 200. The planner resolves the repo by the provider's numeric id, fetches the workflow file at the commit sha, stores it content-addressed, and in one transaction inserts the run, all its steps (step 1 queued, the rest blocked) and a run_created event.
2. Release the next step. The scheduler takes queued steps, checks the tenant's concurrency budget, creates an attempt with a lease, and pushes a dispatch message to that tenant's queue. When step k succeeds, the same transaction that records the success flips step k+1 to queued.
3. Execute. The runner controller turns a dispatch into a pod (or claims a warm one), injects a short-lived job token, and the in-pod agent clones the repo at the sha, runs the command, streams logs, heartbeats every 10 s, and calls Complete with its lease token.
4. Live view. Every state change writes a run_events row; a change-data-capture feed publishes it to the event bus; the live gateway fans it out to SSE subscribers of that run. Log chunks flow on a separate channel straight from the runner API.
5. Deploy. A deploy step hands an artifact digest to the rollout controller, which owns the canary stages and rollback; the step stays running until the rollout finishes or fails.

Deep dives

From push to run: dedupe, repo identity, config snapshot. Git hosts deliver at least once and retry on any slow response, so ingest does the minimum: verify, durably record, ack. Dedupe happens twice: on delivery_id (same delivery retried) and on the run's unique key (repo_id, commit_sha, workflow, trigger_key) (two different deliveries for the same push, e.g. a redelivery from the UI). Resolve the repo by the host's immutable numeric repo id and installation id, never by owner/name: a renamed or transferred repo keeps its history, and a new repo reusing an old name can't inherit its secrets. Read the config at the commit sha, not the branch head, or a fast follow-up push changes what the first run executes. Store the parsed, validated definition by digest and point the run at it; re-runs replay the same definition even if the file is later edited. If the Git host API is rate-limited or down, the plan job retries with backoff; the delivery row proves we owe a run. Validate the YAML against a schema at plan time and cap steps (200), timeout (6 h) and matrix size; a bad file fails the run with a readable error instead of crashing the planner.

Linear scheduling: exactly one next step becomes runnable. The completion path is a single transaction on the tenant's shard:

UPDATE attempts SET status='succeeded', exit_code=0
  WHERE id=$a AND lease_token=$t AND status='running';      -- 0 rows → stale, ignore
UPDATE steps SET status='succeeded' WHERE id=$s AND status='running';
UPDATE steps SET status='queued', ready_at=now()
  WHERE run_id=$r AND ordinal=$k+1 AND status='blocked';
INSERT INTO run_events (run_id, seq, ...) VALUES ($r, run.seq+1, ...);

The conditional updates make duplicates harmless: a retried Complete or a zombie runner whose lease was reassigned matches zero rows. Dispatch uses SELECT ... FOR UPDATE SKIP LOCKED over queued steps so many scheduler replicas can pull without double-leasing, and the partial unique index on live attempts is the backstop. Failure: if n < max_attempts and the failure is retryable (infra error, lease expiry, exit codes listed in retry_on), insert attempt n+1 with exponential backoff in ready_at; otherwise mark the step failed, the remaining steps skipped, and the run failed. Cancellation sets a cancel_requested flag; the agent sees it in its next heartbeat response and gets SIGTERM, then SIGKILL after 30 s. A reaper scans attempts where lease_expires_at < now() (lease = 3 missed 10 s heartbeats), marks them lost, kills the pod and schedules a retry. Fairness: each tenant has a concurrency limit by plan, and dispatch picks the next tenant by deficit round-robin over active tenants, so a 5,000-job burst drains at its share while others keep starting in seconds.

Tenant-safe execution on Kubernetes. Untrusted code runs with root inside its container, so containers alone are not the boundary.

Pick Kata or gVisor runtime classes on multi-tenant pools, one pod per attempt, destroyed after. Pods get no service-account token, egress through a proxy that blocks the cloud metadata endpoint and internal ranges, and a per-attempt OIDC token (audience-scoped, 1-hour TTL) that the secrets service exchanges for only that repo's secrets. Secrets are masked in log output. Caches are keyed tenant/repo/branch/key-hash so a fork PR can read the main branch cache but never write it; fork PRs get no secrets at all. Keep warm pools per image class (e.g. 500 idle ubuntu-large pods) sized from the last hour's arrival rate to hit the 10 s start target, and scale nodes with the cluster autoscaler for the rest.

Live progress in the browser. The page loads GET /v1/runs/{id} (status + version), then opens SSE with after_seq=version. Events are small (step 4 → running), carry seq, and the client applies them only if seq = last + 1; a gap triggers a snapshot re-fetch. The live gateway keeps the last 1,000 events per active run in Redis streams, so most reconnects resume without touching the database. Logs are a separate stream per attempt: the runner API appends chunks to Redis (hot tail, last 5 MB) and flushes 1 MB objects to object storage; the viewer reads stored chunks by offset, then tails. Fan-out: 100k concurrent viewers is fine with ~50 gateway pods holding 2k connections each, subscribed by run id. If streaming breaks, the UI falls back to polling the snapshot every 5 s.

Safe rollout and rollback. A release is an immutable artifact digest plus config version. The rollout controller moves it through stages (1 canary pod → 10% → 50% → 100%), each with a bake time (10 min) and automated gates comparing canary vs baseline: error rate, p99 latency, crash loops. A failed gate rolls back automatically by pointing traffic at the previous release, which is still deployed and warm, so rollback takes seconds and needs no rebuild. Only one deployment per (service, environment) is active; a newer one queues or supersedes. Database migrations must be expand/contract so the old release can still run after rollback.

Trade-offs

• A relational, tenant-sharded control plane gives transactional step transitions; it gives up the near-infinite write scale of a log-based design, so events are partitioned by time and pruned.
• Freezing the definition at the commit gives reproducible re-runs; it means fixing a broken pipeline file always needs a new commit.
• Ephemeral pod per attempt gives clean isolation and no state leaks; it pays in start latency and image pulls, which warm pools and node-local image caches buy back at idle cost.
• Sandboxed runtimes (gVisor/Kata) protect the fleet from tenant code at a 10–30% CPU tax; trusted internal tenants can opt out.
• Per-tenant queues with weighted fair share stop noisy neighbors but leave capacity idle when one tenant's quota is full and others are quiet; allow borrowing up to 2× when the fleet is under 70% busy.
• Separate log and status channels keep builds running when streaming fails, but the UI can briefly show a step finished before its last log lines arrive.
• Webhook ingest runs in several regions and needs only object storage plus the delivery table, so it keeps accepting during a control-plane failover; the control-plane DB fails over asynchronously to a standby region (RPO seconds) and the planner replays unprocessed deliveries, giving up a few seconds of step state that leases and retries repair.
• Logs move to cold storage after 30 days and are deleted at 90: cheaper by ~5×, but old failures are slower to open.
• Automatic rollback on metric gates catches most bad deploys within the bake time but adds 30–40 minutes to a full rollout; hotfixes can skip stages with an audited override.

Interviewer expectations

Mid-level (L4)

• Webhook → queue → workers → status in a DB, with logs in object storage.
• Retries with a limit and timeouts on steps; knows webhooks can arrive twice.
• A basic rollout story: deploy to a few hosts first, roll back if errors spike.

Senior (L5)

• Idempotent ingest with two dedupe keys, config read at the commit sha and snapshotted.
• Leases, heartbeats and conditional updates so a step runs once and a dead worker is recovered.
• Real tenant isolation (sandboxed runtime, scoped secrets, fork PR rules) and live progress with sequence numbers and resume.

Staff (L6+)

• Sizes the fleet and logs and lets cost drive warm pools, autoscaling and retention.
• States the invariants and enforces them in the schema; explains fair-share scheduling across tenants.
• Covers failure domains end to end: regional ingress, control-plane failover with replay, and rollouts that stay reversible including database migrations.