Design a customer support ticketing system

Source: https://www.hack2hire.com/question-bank/questions/system-design/69d1348a17b72775cc49be7b
Company: Airbnb
Difficulty: Medium

Problem statement

Customers open tickets by email, chat or web; agents get them routed by skill and priority, with SLAs, escalation and full history.

Requirements

We are building the support desk behind a marketplace like Airbnb: guests and hosts contact support by email, live chat or a web form; each contact becomes a ticket that is routed to an agent with the right skill at the right priority, tracked against SLAs, escalated when it slips, and kept with a complete, auditable history.

Functional requirements

• Intake from three channels (inbound email, chat widget, web/app form) into one ticket model; a reply on any channel lands on the same ticket.
• Routing: classify each ticket (language, topic such as payments, safety, cancellations; customer tier; booking urgency) and place it in the queue of agents who hold that skill, ordered by priority and SLA deadline.
• Agents pull work: "get next" from their queues, or claim a specific ticket from a shared view. Exactly one owner per ticket at a time.
• Ticket timeline: customer messages, agent replies, internal notes, attachments, and system events (assigned, status changed, SLA breached) in one ordered stream.
• Lifecycle: new, open, pending (waiting on customer), on hold (waiting on a third party), solved, closed; a customer reply reopens a solved ticket within a window.
• SLAs: first-response and next-response deadlines per priority; warnings at 75% and escalation (priority bump, reassign, page a lead) at breach.
• Full audit: who changed what and when, never edited in place.

Non-functional requirements

• No lost contacts: an accepted email or chat message is durable before we acknowledge it; duplicates from vendor retries collapse to one message.
• Claim is exclusive: two agents never both work the same ticket.
• Agent views refresh within ~2 s; "get next" p99 under 200 ms.
• Outbound replies are delivered at least once and shown with their delivery status; they never silently vanish.
• 99.95% availability for intake; routing may lag briefly but intake must never stop.
• PII (addresses, payment details, IDs in attachments) is access-controlled, redacted from search, and deleted on retention schedules.

Back-of-the-envelope estimation

• 150M users; ~1% contact support per month → 1.5M new tickets/month ≈ 50k/day, peaks of 5x during holiday travel or outages ≈ 3 tickets/s, bursts to 30/s.
• ~8 messages per ticket → 400k messages/day ≈ 5/s average, 50/s peak. Small writes; the hard part is correctness, not throughput.
• 10k agents online at peak, each view polling or subscribed: 10k sessions × 1 refresh/2 s = 5k view reads/s, served from a cache, not the ticket DB.
• Storage: 2 KB per message body × 150M messages/year ≈ 300 GB/year of text; attachments ~20% of tickets × 1 MB ≈ 4 TB/year in object storage.

API design

Intake (internal, called by channel adapters):

• POST /v1/intake with {channel: "email"|"chat"|"web", external_id, thread_ref, from: {email, user_id?}, subject?, body, attachments: [{blob_id}], received_at} → {ticket_id, message_id, created: bool}. external_id (email Message-ID, chat vendor message id, form submission id) is the idempotency key; a repeat returns the original ids with created: false.

Agent API:

• GET /v1/views/{view_id}?cursor= → {tickets: [{id, subject, priority, sla_due_at, status, assignee}], cursor, version}.
• POST /v1/queues/{queue_id}/next → {ticket_id, lease_expires_at} or 204 if empty.
• POST /v1/tickets/{id}/claim with {expected_version} → 200 {assignee, version} or 409 {code: "already_claimed", assignee}.
• POST /v1/tickets/{id}/messages with {type: "reply"|"note", body, attachments, client_msg_id} → {message_id, delivery: "queued"}.
• PATCH /v1/tickets/{id} with {status?, priority?, tags?, expected_version} → new ticket, or 409 {code: "stale", current}; illegal transitions return 422 {code: "invalid_transition"}.
• GET /v1/tickets/{id}/timeline?after_seq= → ordered events with seq; clients resume from the last seq they saw.

Errors and retries: every write takes a client-generated key (client_msg_id, external_id) so retries are safe. 409 means re-read and decide; 429 and 503 are retryable with backoff. Auth: agents use SSO sessions with role scopes (tier 1, payments, trust and safety); customers see only their own tickets via the product's session; adapters use service credentials.

Data model

The ticket core lives in a relational store (Postgres/MySQL, sharded by ticket_id) because claims, state transitions and SLA updates need row-level transactions and conditional updates.

Message bodies over 4 KB and all attachments go to object storage, encrypted, referenced by body_ref/blob_id. Search (agent full-text lookup) is a separate index fed from the outbox, with PII fields redacted.

High-level design

flowchart LR
  EM[Email provider] --> AD[Channel adapters]
  CH[Chat vendor] --> AD
  WF[Web form] --> AD
  AD --> IN[Intake service]
  IN --> DB[(Ticket DB)]
  IN --> OB[Outbox relay]
  OB --> K[Event log]
  K --> RT[Routing service]
  RT --> DB
  K --> VW[View cache]
  AG[Agent console] --> API[Ticket API]
  API --> DB
  API --> VW
  K --> DL[Delivery workers]
  DL --> EM
  DL --> CH
  K --> SLA[SLA scheduler]
  SLA --> DB
  K --> SR[Search index]

1. Ingest and route. The email webhook (or chat vendor callback, or form post) hits an adapter that stores attachments in object storage and calls intake. Intake resolves the customer identity, finds an open ticket on the same thread (email In-Reply-To, chat conversation id, or the customer's recent open ticket on the same booking) or creates one, inserts the message with its unique external_id, writes an outbox row, and commits. Only then does the adapter return 200 to the vendor. Routing consumes ticket.created, runs a classifier (topic, language, urgency), applies rules (safety keywords → urgent trust-and-safety queue; top-tier hosts → priority queue), and sets queue_id, priority and SLA due times.
2. Claim. An agent opens a view (a saved filter over a queue), served from the view cache. Clicking a ticket sends claim with expected_version; the DB does UPDATE tickets SET assignee_id=?, version=version+1 WHERE id=? AND version=? AND assignee_id IS NULL. Zero rows → 409. "Get next" does the same against the head of the queue.
3. Messages. An agent reply is inserted as a message with delivery_status=queued plus an outbox row in one transaction. Delivery workers send it via the email or chat vendor and update the status to sent, then delivered/bounced on vendor callbacks. The console streams new timeline events by seq.
4. Resolve, wait, reopen. Status changes go through the state machine in the ticket API, append a ticket_event, and start or stop SLA timers. A customer reply on a solved ticket within 7 days reopens it to the previous assignee; after closure it creates a linked follow-up ticket.

Deep dives

Multi-channel intake and idempotency. Every vendor retries: email providers redeliver on timeouts, chat vendors resend webhooks for up to 24 h. The unique index on (channel, external_id) makes a second insert fail, and intake returns the stored ids. Threading is the harder half. Email uses In-Reply-To/References headers plus a ticket token in the reply-to address (support+t_8f3a@) because many clients strip headers. Chat uses the vendor conversation id. When a customer emails after a chat about the same booking, identity resolution maps both to one customer_id, and a merge rule attaches the email to an open ticket on that booking within 48 h; otherwise it opens a new ticket and links it. Merges are events, not rewrites, so they can be undone.

Routing views, hot queues and claim races. Views are not DB queries per refresh. The view cache keeps per-queue sorted sets keyed by (priority, sla_due_at), updated from the event log, and pushes deltas to consoles over a websocket. With 500 agents on one hot queue (a payments outage), "everyone grabs the top ticket" causes claim storms. Options:

Use "get next" by default and keep browse-and-claim for leads. Popping uses SELECT ... FOR UPDATE SKIP LOCKED on the queue index, so concurrent pops take different rows. A lease that expires without a reply returns the ticket to the queue. For fairness, an agent's next pick is weighted across their skills by SLA slack, so one flooding queue does not starve the rest; idle agents in adjacent skills can steal from a queue whose oldest ticket exceeds 50% of its SLA.

Conversation delivery and recovery. Outbound uses the transactional outbox: the reply and its outbox row commit together, so a crash cannot lose a reply that the agent saw as saved. Delivery workers send with the message id as the vendor idempotency key (or the email Message-ID), retry with backoff for 24 h, and mark failed after that, which raises the ticket to the agent with a "not delivered" banner and an option to switch channel (chat went offline → send by email). Consoles and customer apps recover by reading timeline?after_seq= after any disconnect, so missed pushes never cause gaps.

State machine, audit and reopen correctness. Allowed transitions live in one table in code: new→open, open→pending|on_hold|solved, pending→open (on customer reply) or pending→solved (after 5 days without reply), solved→open (reply within 7 days), solved→closed (after 7 days). Every transition is a conditional update on version plus an append to ticket_events in the same transaction; the tickets row is a projection, the event log is the truth. Reopen keeps the original ticket id, increments reopen_count (a quality metric), restores the last assignee if they are online, and starts a fresh next-response SLA. A reply arriving after closed creates a new ticket linked by parent_ticket_id, so closed metrics never change.

SLA timers and escalation. Each ticket has at most a few live timers in sla_timers. A scheduler, partitioned by ticket_id hash across workers, scans (state=active, due_at < now() + 1 min) every 10 s and enqueues warning or breach events; firing is idempotent because the event writes state=fired with a conditional update. Timers pause in pending and on_hold (time is counted against us only when we owe a response) and stop on the first public reply. Escalation policies are data: at breach, bump priority one level and notify the queue lead; at 2x breach, move to the escalations queue. Business hours per queue and timezone are applied when computing due_at.

Trade-offs

• A relational core with conditional updates gives exact claims and transitions but needs sharding work beyond a few thousand writes per second; this load is far below that.
• Event log as truth plus a projected tickets row costs extra writes and a rebuild path, but gives a complete audit and safe replays.
• "Get next" with leases removes claim races but takes choice away from agents; leads keep browse-and-claim for judgement calls.
• Auto-merging contacts across channels cuts duplicate work but can merge two different problems; we only auto-merge within one booking and 48 h, and merges are reversible.
• Reopen-in-place keeps the conversation together but makes "solved" counts move; closing after 7 days freezes reporting.
• Out of scope: the classifier model itself, agent workforce scheduling, the help-center/self-service bot, and phone telephony.

PII handling: attachments are scanned and stored encrypted with per-region keys; agents get short-lived signed URLs; message bodies are retained 2 years (longer under legal hold), then bodies and attachments are deleted while the event skeleton (timestamps, statuses) stays for metrics.

Interviewer expectations

Mid-level (L4)

• Clear ticket, message and assignment model with a sensible status list.
• Working intake for each channel and a basic queue by priority.
• Notices that two agents can claim the same ticket and fixes it with a conditional update.

Senior (L5)

• Idempotent intake keyed on vendor message ids and reliable email threading.
• Outbox-based delivery with retries, delivery status, and resumable timelines.
• An explicit state machine with audit events, SLA timers that pause correctly, and reopen rules.
• Queue design that avoids claim storms (atomic pop, leases).

Staff (L6+)

• Fairness across hot and cold queues, work stealing, and escalation as policy data.
• Identity resolution and cross-channel merge with reversible links.
• PII boundaries, retention and deletion that keep metrics intact.
• Says what they would not build, and how the design changes at 10x volume or during a major outage.