Design an AI prompt playground

Question: Design an AI prompt playground

Source: https://www.hack2hire.com/question-bank/questions/system-design/69dbcc73e4441834b3f72fad

Companies: Anthropic

Difficulty: Medium

Problem statement

Design a web workspace where developers write prompts, run them against several models and settings, compare outputs side by side, and save and share versions.

Requirements

We are building a web workspace where developers write a prompt (system text, messages, variables, tools), run it against several models and parameter settings at once, watch the outputs stream in side by side, and save, diff, revert, fork and share versions of that prompt.

Clarifying questions worth asking first: Is this our own model API or third-party providers too (assume both, behind one adapter)? Do runs bill against the developer's API quota (yes)? Is sharing within an org or public links (both; public is read-only)? Do we keep every run's output forever (90 days by default, pinned runs forever)?

Functional requirements

• Edit a prompt: system prompt, message list, {{variables}} with test values, tool definitions, attachments (files, images).
• Run one prompt against up to 6 "columns" (model + temperature, max_tokens, top_p, etc.) in parallel; outputs stream token by token.
• Save a version explicitly or auto-save drafts; list history, diff any two versions, revert (revert creates a new version, never rewrites history).
• Compare: pick two versions or two runs and see prompt diff plus output diff, latency, tokens and cost.
• Fork a prompt (yours or a shared one) into your own workspace; keep lineage.
• Share with org members (view/edit) or via a read-only link; export as API code.
• Batch run a prompt over a small dataset of variable rows (up to ~100 rows) and attach scores.

Non-functional requirements

• First token visible within the model's own TTFT + under 150 ms of our overhead.
• A refresh or network blip mid-stream must not lose output or re-bill the run.
• Versions are immutable and durable (11 nines on blob storage); runs are best effort after 90 days.
• Editor feels local: keystrokes never wait on the server; saves are async.
• Cost safety: no single user or bug can burn unbounded tokens.
• Prompts can be large (200K-token context, multi-MB attachments) without slowing list pages.

Back-of-the-envelope estimation

• 500K monthly developers, 50K daily active; peak 2K concurrent active sessions per 10K DAU → ~10K concurrent editors.
• Runs: 50K DAU × 40 runs/day × 2.5 columns ≈ 5M model calls/day ≈ 60/s average, ~300/s peak.
• Each call streams for ~10 s → ~3K concurrent open streams at peak. Easy for a few dozen stream gateway pods (10K–50K connections each).
• Prompt text: median 4 KB, p99 400 KB, max ~1 MB of text plus attachments up to 30 MB.
• Versions: 50K DAU × 15 saves/day = 750K versions/day. Content-addressed dedupe cuts that roughly in half; at 8 KB average → ~3 GB/day of blobs, ~1 TB/year. Metadata rows ~300 B → ~80 GB/year.
• Run outputs: 5M/day × 2 KB ≈ 10 GB/day; 90-day retention ≈ 1 TB in object storage.

API design

POST /v1/prompts                     {name, workspace_id} -> {prompt_id}
PUT  /v1/prompts/{id}/draft          {base_version_id, body, client_rev} -> {client_rev}
POST /v1/prompts/{id}/versions       {base_version_id, message, body_hash | body}
     -> 201 {version_id, number: 42, body_hash}
     -> 409 {code: "stale_base", head_version_id}       # someone saved in between
GET  /v1/prompts/{id}/versions?cursor=&limit=50
     -> {items: [{version_id, number, author, message, created_at, body_hash, size}], next_cursor}
GET  /v1/versions/{vid}              -> {body inline if < 64 KB, else body_url (signed, 15 min)}
POST /v1/prompts/{id}/revert         {to_version_id, base_version_id} -> 201 {version_id, number: 43, reverted_from: 17}
POST /v1/prompts/{id}/fork           {version_id, target_workspace_id} -> {prompt_id}
GET  /v1/versions/{a}/diff/{b}       -> {hunks: [...]}

POST /v1/runs                        Idempotency-Key: <uuid>
     {version_id | draft_body_hash, variables: {...},
      columns: [{model: "model-a", params: {temperature: 0.7, max_tokens: 1024}}, ...]}
     -> 202 {run_id, column_runs: [{column_run_id, status: "queued"}], stream_url}
GET  /v1/runs/{run_id}/stream?cursor=<column_run_id:seq,...>   (SSE)
     event: delta  data: {column_run_id, seq: 118, text: "..."}
     event: done   data: {column_run_id, stop_reason, usage: {in: 812, out: 340}, cost_usd, latency_ms}
POST /v1/runs/{run_id}/cancel
POST /v1/column_runs/{id}/scores     {scorer: "human" | "rubric:v3", value: 4, comment}

Compact interfaces internally: ModelAdapter.stream(model, messages, params) -> AsyncIterator<Delta>, BlobStore.put(hash, bytes), Quota.reserve(user, est_tokens) -> lease.

Data model

Metadata in Postgres (relational: workspaces, ACLs, version chains, transactional head pointer). Bodies in object storage, content addressed. Stream buffers in Redis.

Query stories that justify the indexes: open a prompt (prompts by id → head version → blob), page history (versions by prompt_id, created_at), compare two versions (two blob fetches by hash), "my recent runs on this prompt" (runs by prompt_id, created_at), "shared with me" (shares by principal). Runs table partitioned by month so 90-day expiry is a partition drop.

Redis: stream:{column_run_id} as a Redis Stream of deltas, TTL 1 hour after done.

High-level design

flowchart LR
  B[Browser editor] --> API[Playground API]
  B <-->|SSE| SG[Stream gateway]
  API --> PG[(Postgres metadata)]
  API --> OBJ[(Object storage blobs)]
  API --> Q[Quota and cost service]
  API --> RQ[Run queue]
  RQ --> W[Run workers]
  W --> MA[Model adapters]
  MA --> M1[Model provider A]
  MA --> M2[Model provider B]
  W --> RS[(Redis streams)]
  SG --> RS
  W --> OBJ
  W --> PG
  W --> EV[Eval scorers]

1. Edit and run. The editor keeps state locally and autosaves the draft every few seconds (PUT draft, body uploaded once per new hash). Clicking Run sends POST /v1/runs with an idempotency key, the version or draft hash, variables and columns.
2. The API resolves the body from blob storage, renders variables, estimates input tokens, and reserves quota for in + max_tokens per column. Over budget → 429 with which limit was hit.
3. It writes runs and column_runs rows and enqueues one job per column. It returns stream_url.
4. A worker calls the model adapter with streaming on and appends each delta to stream:{column_run_id} with a sequence number. The browser's SSE connection to the stream gateway reads all columns' Redis streams and multiplexes them.
5. On completion the worker writes the full output to object storage, records usage and cost, settles the quota lease (refunds unused max_tokens), and fires eval hooks.
6. Browse and compare. History pages read prompt_versions metadata only. Selecting two versions fetches both bodies by hash (CDN-cached; immutable) and diffs on the client; output comparison loads the latest column runs for each version.
7. Save and revert. Save is compare-and-set on prompts.head_version_id; revert writes a new version whose body_hash equals the target's, so no bytes are copied.

Deep dives

Large prompts and the storage split. Prompt bodies range from 1 KB to multiple MB, and the same body gets saved, run and forked many times. Keep only metadata in Postgres and put the canonical JSON body in object storage keyed by SHA-256. Benefits: list and history pages stay small (a row is ~300 B regardless of prompt size); dedupe is free (revert, fork, re-save of an unchanged prompt add zero bytes); bodies are immutable, so they cache forever at the CDN and in the browser. Uploads: the client hashes the body, calls HEAD /blobs/{hash}, and uploads only if missing (presigned PUT for anything over 64 KB). Attachments are separate blobs referenced by hash from the body, so editing the system prompt doesn't re-upload a 20 MB PDF. Bodies under 64 KB are also inlined in the version response to save a round trip on the common case. Garbage collection uses refcounts from versions, drafts and pinned runs; a blob is deleted 7 days after refcount hits zero. For diffs of huge prompts, diff per message block rather than the whole string so it stays under 50 ms.

Streaming delivery and reconnect. Do not couple the model call to the browser's connection. The worker owns the model stream and writes to a Redis Stream; the browser is just a reader. That gives:

The client tracks the last seq per column and reconnects with ?cursor= (or SSE Last-Event-ID); the gateway replays from Redis with XREAD and then tails. Deltas are coalesced to ~30 ms batches to cut event count by about 5× without visible lag. Stream keys live 1 hour after completion; after that the client loads the final output from object storage. If the browser disconnects and nobody reconnects for 2 minutes, the worker keeps going (the user paid for it) unless the user hits Cancel, which sets a flag the worker checks between chunks and aborts the upstream request. Idempotency key on POST /runs makes a double click or a retried request return the same run_id.

Versioning: drafts, saves, reverts, concurrent editors. Separate a mutable per-user draft from immutable versions. Drafts autosave freely and are last-write-wins per user (client_rev rejects out-of-order writes from the same tab). A save creates version N+1 with parent_version_id = base_version_id inside one transaction: UPDATE prompts SET head_version_id = :new WHERE id = :id AND head_version_id = :base. Zero rows → 409 stale_base with the new head, and the UI shows a three-way diff (base, theirs, mine) for the user to merge. Revert is just a save whose body hash is the old version's. History is linear per prompt with fork edges across prompts; real-time co-editing (CRDT) is out of scope unless the interviewer asks, since prompts are usually edited by one person at a time.

Fan-out runs, rate limits and cost protection. One click can fire 6 columns × 100 dataset rows = 600 model calls of up to 200K tokens each. Protections layered from cheap to strict:

• Per-user token bucket on run creation (e.g. 20 runs/min) and a cap on concurrent column runs (e.g. 12), enforced at the API.
• Pre-flight cost estimate: input_tokens × in_price + max_tokens × out_price per column; show it, and require confirmation over $1 or for batch runs.
• Quota reservation: reserve the worst case from the user's daily/org budget in Redis (atomic DECRBY), settle with real usage on completion. Prevents 600 in-flight calls overshooting the budget.
• Per-model concurrency pools in the workers so one popular model's provider limits don't starve others; queue with backpressure and show "queued" in the column.
• Cache identical runs: key = hash(body, variables, model, params) when temperature = 0; offer "reuse output" instead of paying twice.

Evaluation hooks and scoring. After each column run completes, an event column_run.completed goes on a bus. Scorers subscribe: deterministic checks (JSON schema valid, regex, length), a rubric scored by a grader model, and human thumbs/1–5 ratings in the UI. Scores are stored per column run with the scorer's version so results are comparable over time. In batch mode the compare view aggregates per version: mean score, pass rate, p50/p95 latency, total cost, so "v42 vs v41" is a table, not eyeballing. Grader calls go through the same quota system and are billed to the user.

Trade-offs

• Content-addressed blobs give dedupe, caching and cheap reverts, but cost an extra round trip on first load and need refcount GC; inlining bodies under 64 KB hides most of that.
• Worker-owned streams via Redis add a hop (~2–5 ms per batch) and a Redis dependency, in exchange for resumable streams and no re-billing on refresh.
• Optimistic concurrency with 409 + merge is simpler than CRDT co-editing, but two people editing the same prompt at once get a merge dialog.
• Reserving worst-case quota protects budgets but can block a user who has headroom in practice; settlement refunds it within seconds.
• Keeping runs only 90 days bounds storage at ~1 TB but loses old comparisons unless the user pins them.
• Caching temperature-0 outputs saves cost but can hide provider-side model changes; label cached outputs clearly.

Interviewer expectations

Mid-level (L4)

• Clear CRUD for prompts and versions, a run endpoint, and parallel calls to several models.
• Knows outputs should stream and that versions should be immutable rows rather than overwritten text.
• Basic rate limiting per user.

Senior (L5)

• Splits metadata from large bodies, uses content hashing for dedupe and cheap revert/fork.
• Decouples the model call from the browser connection; designs resumable streaming with sequence numbers and idempotent run creation.
• Optimistic concurrency on save with a clear conflict story; indexes justified by actual query paths.
• Quota reservation and cost estimates before fan-out runs.

Staff (L6+)

• Frames the product around the iterate-compare loop: batch runs over datasets, eval hooks, and aggregated version-vs-version reports as the real value.
• Plans for multiple providers with different rate limits, per-model worker pools, and provider outages surfaced per column.
• Owns cost as a first-class system concern (budgets, caching, settlement) and data lifecycle (retention, pinning, GC).
• Calls out what is deliberately out of scope (real-time co-editing) and when it would be worth adding.