Design a sandboxed cloud IDE

Source: https://www.hack2hire.com/question-bank/questions/system-design/6acc71782e3a0349b3a8cbe2
Company: OpenAI
Difficulty: Medium

Problem statement

Give every user a browser-based editor with a terminal and a private container that runs their code, starts in seconds, isolates untrusted code, and saves their work.

Requirements

We are building a browser-based cloud IDE. Each user gets a workspace: an editor, one or more terminals, and a private Linux sandbox that runs their code. The sandbox starts in a few seconds, keeps untrusted code away from other tenants, and saves the user's files between sessions. Questions to ask first: do workspaces run long-lived servers that need a public preview URL? Is GPU in scope? How long must an idle workspace persist? Is there an SSH or "devbox" mode besides the browser?

Functional requirements

• Create a workspace from a template (language image) or a Git repo, then open or resume it from the browser.
• Edit files, open interactive terminals (a real PTY), and resize them.
• Start a run or build (npm test, make), stream its output live, and read the full log later.
• Expose a port from the sandbox as a private preview URL (https://3000-ws123.preview.example.dev).
• Files and installed packages under /workspace persist across stop and resume. Processes do not.
• Stop on idle; delete on request or after 30 days of inactivity.

Non-functional requirements

• Resume p50 under 2 s, p99 under 8 s; cold create from a template under 10 s.
• Keystroke echo in the terminal under 100 ms round trip within a region.
• Strong isolation: a user who gets root inside the sandbox must not reach the host, other sandboxes, the cloud metadata service, or internal networks.
• Durability of saved work: lose at most ~30 s of writes if a host dies.
• Per-user quotas on CPU, memory, disk, egress and concurrent sandboxes; resist crypto-mining.
• Control plane at 99.95%; one host failing affects only the sandboxes on it.

Back-of-the-envelope estimation

• 5M registered users, 500k daily actives, 100k concurrent sandboxes at peak.
• Sandbox shape: 2 vCPU / 4 GB limit, but measured average use is ~0.3 vCPU and 1.5 GB. Hosts are 64 vCPU / 256 GB bare metal. With 3× CPU overcommit and memory packed to 80%: ~100 sandboxes per host is CPU-bound (64 × 3 / 2), ~135 memory-bound → plan ~90 per host → ~1,100 hosts at peak plus 15% headroom and warm pool ≈ 1,300.
• Storage: 10M workspaces × median 400 MB of user data (base image layers shared and not counted) ≈ 4 PB in object storage. Active disks on hosts: 100k × 2 GB ≈ 200 TB local NVMe.
• Session starts: 500k DAU × 4 resumes/day ≈ 2M/day ≈ 25/s average, ~150/s at the 9 a.m. peak.
• Terminal traffic is tiny (a few KB/s per terminal) but long-lived: ~150k concurrent WebSockets across the gateways.
• Run logs: 1M runs/day × median 200 KB ≈ 200 GB/day, kept 14 days ≈ 3 TB.

API design

Browser-facing REST plus WebSockets. Auth is a user session; the gateway mints short-lived, sandbox-scoped tokens.

• POST /v1/workspaces {template_id | repo_url, name} → 201 {workspace_id, state: "creating"}
• POST /v1/workspaces/{id}/sessions (open or resume; idempotent per client) → 200 {session_id, state: "starting" | "running", sandbox_endpoint, connect_token, expires_at}. The client polls GET /v1/sessions/{id} or listens on the session socket for running.
• DELETE /v1/sessions/{id} → explicit stop (snapshot, then release the sandbox).
• GET /v1/workspaces/{id}/files?path=src/ and PUT /v1/workspaces/{id}/files?path=src/app.js (body = content, If-Match: <etag>) → proxied to the in-sandbox agent.
• WS /v1/sessions/{id}/terminals?cols=120&rows=40 → binary frames: stdin, stdout, resize {cols, rows}, exit {code}. Reconnect with terminal_id to reattach to the same PTY and replay the last 64 KB.
• POST /v1/workspaces/{id}/runs {command: "npm test", env: {...}} → 202 {run_id, state: "queued"}
• GET /v1/runs/{run_id} → {state: "running" | "succeeded" | "failed" | "killed", exit_code, started_at, ended_at}
• GET /v1/runs/{run_id}/log?from_offset=0 → chunked text, with X-Next-Offset; WS /v1/runs/{run_id}/log/stream?from_offset=N for live tail.
• POST /v1/workspaces/{id}/ports {port: 3000, visibility: "private"} → {preview_url}.
• Internal (control plane → host agent, gRPC): StartSandbox(sandbox_id, image, snapshot_ref, limits), Snapshot(sandbox_id) → snapshot_ref, StopSandbox, Heartbeat(host_id, sandboxes[]).

Data model

Control-plane metadata lives in Postgres (sharded by user_id once large); it is small, relational and needs transactions on state changes.

• Workspace snapshots in object storage (S3/GCS): content-addressed chunks chunks/{sha256} plus a manifest snapshots/{workspace_id}/{seq}.json listing chunk hashes. Dedup across users makes node_modules almost free.
• Base images as read-only OCI layers or ext4 rootfs images, cached on every host's NVMe.
• Run logs: live in Redis Streams log:{run_id} (capped, TTL 1 h); on completion flushed to logs/{run_id} in object storage, 14-day lifecycle.
• Routing table sandbox_id → host:port in Redis for the gateways, written by the scheduler, with the Postgres row as source of truth.

High-level design

flowchart LR
  B[Browser IDE] --> GW[Edge gateway WebSocket proxy]
  GW --> API[Workspace API]
  API --> PG[(Postgres metadata)]
  API --> SCH[Scheduler]
  SCH --> HA[Host agent]
  GW --> HA
  HA --> VM["Sandbox microVM"]
  VM --> IA[In-sandbox agent]
  HA --> OS[(Object storage snapshots and logs)]
  IA --> RS[(Redis log streams)]
  GW --> RS
  HA -. heartbeat .-> SCH
  VM --> EG[Egress proxy]

The control plane (API, scheduler, Postgres, reaper) decides where sandboxes live and tracks state. The data plane (gateways, host agents, microVMs, egress proxy) carries bytes. User traffic never goes through the control plane after connect, so a control-plane outage stops new starts but does not drop open terminals.

1. Open or resume. POST /sessions → API checks quota and looks for a live sandbox (unique index). None → scheduler picks a host: prefer one that still has this workspace's disk cached locally, else the least-loaded host in the user's region with the base image present. Host agent claims a pre-booted warm microVM, attaches the workspace disk (local cache hit, or lazy-fetched from the latest snapshot), and starts the in-sandbox agent. Agent reports running; the scheduler writes the route; the browser connects.
2. Terminal. Browser opens a WebSocket to the gateway with the connect token. The gateway checks it, looks up sandbox_id → host, and opens a stream to the host agent, which forwards over vsock to the in-sandbox agent. The agent spawns a shell on a PTY and pipes bytes both ways. The PTY outlives the socket for 10 minutes so a network blip reattaches to the same shell.
3. Run with live logs. POST /runs → API writes the row and calls the in-sandbox agent, which starts the command in its own process group and cgroup. The agent appends stdout/stderr chunks with offsets to log:{run_id}; viewers tail from any offset through the gateway. On exit the agent records the code, the host agent uploads the full log to object storage, and the row flips to a terminal state.
4. Idle and stop. The agent reports activity (keystrokes, running processes, open ports). After 30 min idle (or on explicit stop) the host agent flushes a final snapshot, marks the workspace stopped with the new latest_snapshot_ref, and returns the VM slot.

Deep dives

Sandbox lifecycle and persistence: making resume take 2 seconds. The slow parts are booting a kernel, pulling the image, and restoring user files. Attack each:

• Warm pool. Each host keeps ~5% of slots as booted, generic microVMs per popular base image (Firecracker boots in ~125 ms, but runtime init takes longer). A start claims one and hot-attaches the workspace disk. The pool is refilled in the background; pool size per image tracks the last hour's start rate.
• Disk layout. Root filesystem = read-only base image + a per-workspace writable overlay. /workspace and the overlay live on a block device backed by local NVMe. Every 30 s and on stop, the host agent takes a copy-on-write snapshot, chunks changed blocks (4 MB, content-hashed), and uploads only new chunks. A typical incremental snapshot is a few MB.
• Lazy restore. On a host without the disk cached, mount the snapshot through a block driver that fetches chunks on first read, prefetching the chunks read in the first 10 s of the last session. The shell is usable in ~1 s while the rest streams in.
• Affinity. Re-scheduling a workspace onto the host that ran it last hits the local cache ~70% of the time; caches evict LRU when NVMe passes 80%.
• Memory snapshots (optional). Firecracker can snapshot guest RAM to resume running processes, but 4 GB per workspace × 10M is too much storage. Keep RAM snapshots only for 1 h after stop ("stepped away for lunch"); after that fall back to disk-only.

Run/build execution and log streaming. Runs execute inside the user's sandbox, so they share its limits; each run gets its own child cgroup so "kill run" kills the whole process tree. Logs are an append-only byte stream with offsets: the agent batches output every 50 ms or 16 KB, whichever comes first, and appends {offset, bytes} to the Redis stream. A reader asks for from_offset, gets the backlog, then live entries; a reconnect resumes from its last offset with no gaps or duplicates. Cap a single run at 50 MB of log (truncate the middle and say so) and cap output rate at 1 MB/s per run so yes cannot flood Redis or the browser. Terminals are different: they are interactive and not stored server-side beyond a 64 KB ring buffer for reattach.

Isolation: running strangers' code on shared hosts. Assume every user has root in their sandbox and a kernel exploit in hand.

Layer defenses around the microVM: the VMM runs as an unprivileged user in a jailer (chroot, seccomp, own cgroup); no shared filesystems with the host except a read-only image; vsock is the only host channel and the host agent treats everything on it as untrusted input. Network: each VM gets a tap device on its own VLAN or eBPF-filtered interface; it can reach only the egress proxy, never 169.254.169.254, other VMs, or internal ranges. Preview URLs go through the gateway with the user's auth cookie, so one user cannot browse another's dev server. Hosts are single-purpose, patched by draining, and never hold cloud credentials beyond what the host agent needs.

Host failure and zombie sandboxes. The host agent heartbeats every 5 s with the list of sandboxes it runs. Each sandbox row has a lease renewed by heartbeats. If a host misses heartbeats for 30 s, the scheduler marks it dead, marks its sandboxes lost, and the next open resumes from the latest snapshot on another host (losing at most the last 30 s of writes). To stop a "dead" host that is only partitioned from coming back and writing a stale snapshot, every snapshot upload carries the sandbox generation; the API accepts a manifest only if it matches the current generation (a fencing token). The reverse case, a VM the control plane forgot about, is caught by reconciliation: the host agent compares its running VMs with what the scheduler says it should run every minute and kills strangers.

Abuse: crypto-mining, egress, and quotas. Free compute attracts miners. Controls: CPU quota per sandbox via cgroups (2 vCPU burst, throttled to 0.5 vCPU after 10 min sustained at 100% on free tier); a monthly CPU-hour budget per user; idle stop that counts only real user activity, not a spinning process; egress through a proxy with per-user bandwidth caps (e.g. 5 GB/month free), blocked ports (SMTP 25, known mining pools) and DNS-level denylists; and a detector on host metrics (sustained full CPU, stratum protocol on the wire, known miner binaries by hash) that suspends the account for review. New accounts get lower limits until they verify a phone or payment method.

The SSH / devbox variant and fleet capacity. An SSH front door reuses everything: ssh ws123@ssh.example.dev hits an SSH gateway that authenticates the user's key, calls the same open-session API, and pipes the channel to the same in-sandbox agent. Only the edge protocol changes. For capacity, keep 15% spare across each zone for a zone failure and the warm pool; scale the fleet on forecast plus allocated-slot ratio, since bare-metal hosts take minutes to add. Bin-pack on memory (it cannot be overcommitted safely) and let CPU overcommit absorb bursts; drain the least-loaded hosts at night to save cost.

Trade-offs

• MicroVMs give a hardware boundary for hostile code but cost more per sandbox and need bare-metal or nested-virtualization hosts; gVisor would run on ordinary VMs with weaker compatibility.
• Persisting only the disk (not RAM) keeps storage at ~400 MB per workspace instead of 4+ GB, at the cost of users losing running processes after a stop.
• Snapshotting every 30 s bounds data loss but costs upload bandwidth; a replicated network block device would lose nothing on host failure but adds write latency and cost to every keystroke-driven save.
• The warm pool makes starts fast but keeps ~5% of capacity idle; host affinity speeds resume but makes bin-packing less even.
• CPU overcommit at 3× cuts the fleet by roughly two thirds but means noisy neighbors when many users compile at once; fixed reservations would be predictable and three times the cost.
• Routing terminals through a gateway adds a hop (~5 ms) but keeps hosts off the public internet and lets one place enforce auth and preview-URL access.

Interviewer expectations

Mid-level (L4)

• Separates the browser editor, a per-user container, and persistent storage; describes the WebSocket terminal path.
• Knows plain containers share a kernel and that untrusted code needs more than that.
• Covers idle stop, saving files to durable storage, and per-user resource limits.

Senior (L5)

• Splits control plane from data plane and explains what keeps working when each fails.
• Gets resume to seconds with a warm pool, overlay disks, incremental snapshots and lazy restore, with latency numbers per path.
• Compares containers, gVisor and microVMs; locks down network egress and the metadata endpoint.
• Designs log streaming with offsets for gap-free reconnect, and caps log volume.

Staff (L6+)

• Handles host death and partitions with leases, reconciliation and fenced snapshot writes.
• Sizes the fleet from overcommit ratios and memory bin-packing, and states the cost of the warm pool and zone headroom.
• Treats abuse (mining, egress, signup fraud) as a first-class design input with quotas and detection.
• Shows that SSH, preview URLs and future GPU sandboxes reuse the same lifecycle with a different front door.