Design a real-time air traffic control system

Source: https://www.hack2hire.com/question-bank/questions/system-design/69e124dc95e80ae213e5cfa6
Company: Amazon
Difficulty: Medium

Problem statement

Track every aircraft in a region from radar and transponder feeds, predict conflicts, and alert controllers within a second, with no single point of failure.

Requirements

We are building the surveillance and conflict-alert core for one air traffic control facility: it fuses radar and transponder (Mode S, ADS-B) feeds into one track per aircraft, predicts losses of separation, and puts alerts on the right controller's screen within a second, with no single point of failure. Voice, clearance delivery and flight-plan filing are out of scope; we consume flight plans from an existing system.

Clarifying questions worth asking first: how big is the region (one terminal area or a whole en-route center)? Which separation minima apply (3 nm terminal, 5 nm en route, 1,000 ft vertical)? Is the system advisory (alerts a human) or does it issue clearances? (Advisory.) How long must recordings be kept, and who replays them?

Functional requirements

• Ingest position reports from primary/secondary radars and ADS-B ground stations, each with its own rate, accuracy and latency.
• Fuse reports into one track per aircraft (position, velocity, altitude, identity) and correlate it with its flight plan by callsign or squawk code.
• Stream live tracks to each controller position, filtered to its sector plus a buffer around it.
• Predict conflicts: short-term alerts (0–2 min look-ahead) and a medium-term probe (up to 20 min) for planned routes.
• Alert the controller who owns either aircraft; let them acknowledge or suppress an alert.
• Record every input, track and alert for incident replay, and serve track history by flight and time range.
• Hand off aircraft between sectors (and later between facilities).

Non-functional requirements

• Sensor report to updated track on screen: p99 under 1 s. Conflict detected to alert on screen: p99 under 1 s.
• Availability 99.999% for the surveillance picture (about 5 min/year); no single box, link, rack or power feed can blank a display.
• Correctness over throughput: a missed conflict is a safety event; a false alert costs controller attention, so both are measured.
• Every display must show how fresh each track is; stale data must never look live.
• Audit records are complete and tamper-evident; replay reproduces what the controller saw.

Back-of-the-envelope estimation

• Region: up to 5,000 airborne aircraft at peak, 40 radars (4–12 s rotation), 250 ADS-B stations.
• ADS-B: 5,000 aircraft × 1 report/s × ~4 stations hearing each = 20,000 reports/s. Radar: 5,000 × ~3 radars ÷ 6 s ≈ 2,500 plots/s. Total ~25,000 reports/s at ~200 B ≈ 5 MB/s. Small; the hard part is latency and correctness, not volume.
• Tracks: 5,000 updated at 1 Hz = 5,000 track updates/s.
• Displays: 150 controller positions × ~300 tracks in view at 1 Hz ≈ 45,000 messages/s, ~100 B each ≈ 4.5 MB/s out.
• Conflict pairs: naive all-pairs is 12.5M pairs/s. A spatial grid of 10 nm cells × 2,000 ft bands cuts it to roughly 50,000 candidate pairs/s, each a few microseconds of math.
• Storage: raw reports 25,000/s × 200 B × 86,400 s ≈ 430 GB/day; tracks 5,000/s × 120 B ≈ 52 GB/day. Hot retention 45 days ≈ 22 TB raw; compressed (~5×) to cold storage for years.

API design

Internal gRPC between processes over two independent networks (LAN A and LAN B). Sensors usually speak legacy formats (ASTERIX), so front-end adapters convert at the edge.

// Sensor adapter -> ingest
rpc ReportPositionBatch(Batch) returns (Ack)
Batch { sensor_id, seq_no, sent_at, reports: [
  { sensor_time, kind: RADAR|MODE_S|ADSB, icao24?, squawk?, lat, lon,
    alt_ft, ground_speed_kt?, track_deg?, quality: {nacp, sigma_m} } ] }
Ack { sensor_id, highest_seq_no }

// Display -> track service (server-streaming)
rpc SubscribeTracks(Sub) returns (stream TrackFrame)
Sub { position_id, sector_ids: [..], buffer_nm: 20, resume_from_epoch? }
TrackFrame { epoch, tracks: [ { track_id, flight_id?, callsign, lat, lon,
  alt_ft, vs_fpm, gs_kt, heading, last_update, staleness: FRESH|COAST|DROPPED,
  sources: [..] } ], alerts: [..], snapshot: bool }

rpc GetTrackHistory(flight_id | track_id, from, to, resolution_s) returns (stream TrackPoint)
rpc AcknowledgeAlert(alert_id, position_id, action: ACK|SUPPRESS_60S) returns (Alert)
rpc InitiateHandoff(track_id, from_sector, to_sector) returns (Handoff)
rpc AcceptHandoff(handoff_id, position_id) returns (Handoff)
rpc Health() returns (ComponentStatus)

seq_no per sensor lets ingest detect gaps and duplicates. epoch is a monotonic per-tracker frame counter so a display that reconnects can ask for a fresh snapshot and know it missed nothing older.

Data model

Pair keys sort the two track IDs (min|max) so A–B and B–A are the same conflict.

High-level design

flowchart LR
  R[Radars] --> FE[Front end adapters A and B]
  ADS[ADS-B stations] --> FE
  FE --> LOG[(Replicated report log)]
  LOG --> TA[Tracker primary]
  LOG --> TB[Tracker hot standby]
  FP[(Flight plan DB)] --> TA
  FP --> TB
  TA --> CD[Conflict detection partitions]
  TB --> CD
  CD --> AM[Alert manager]
  TA --> DG[Display gateways]
  TB --> DG
  AM --> DG
  DG --> W[Controller workstations]
  LOG --> REC[Recorder and history store]
  AM --> REC
  FE -. raw plots bypass .-> DG

1. Ingestion and fanout. Each sensor connects to two front-end adapters on separate networks. Adapters timestamp, convert to the common report format, and append to the log; duplicates from the two paths are dropped by (sensor_id, seq_no). Two trackers (primary and hot standby on separate hardware and power) consume the same partitions.
2. Tracking. Each report is associated to an existing track (gating on predicted position) or starts a tentative track. A filter (Kalman with turn/straight models, IMM) updates position and velocity. Tracks with Mode S or ADS-B identity correlate to flight plans. The tracker emits a frame every second per sector region.
3. Conflict detection. Detection is partitioned by spatial cell. Every second it projects each track forward, tests candidate pairs in neighbouring cells, and emits conflict candidates to the alert manager, which deduplicates, applies hysteresis and routes the alert to the owning sectors.
4. Display and recovery. Display gateways hold each workstation's subscription, merge tracks and alerts for its sectors plus buffer, and push deltas. On reconnect, the gateway sends a full snapshot at the current epoch. If both trackers fail, the display falls back to raw radar plots straight from the front end (no tracks, no alerts, clearly flagged), which is how controllers keep a picture during a total tracker outage.
5. Recording. The recorder writes every report, frame and alert to history and seals hourly archive segments.

This holds together because the log is the one ordered truth: trackers are deterministic functions of it, so a standby produces the same tracks, a crashed node rebuilds by replay, and an incident replay reproduces the screen.

Deep dives

Scaling and protecting the ingestion pipeline. Volume (~25k reports/s) fits on one machine, so we partition for isolation and latency, not throughput. Each sensor gets its own partition, so a misbehaving radar can't delay others. Latency budget: adapter 20 ms, log append with RF=3 in-rack acks 10 ms, tracker update 50 ms, conflict cycle 200 ms, gateway push 50 ms, display render 100 ms — about 450 ms, leaving headroom for 1 s. Time is the subtle part: sensors stamp measurement time with GPS-disciplined clocks, and the tracker processes in measurement order with a short reorder window (~200 ms); a report later than that is applied as an out-of-sequence update or dropped and counted. Never use arrival time for physics. If a facility grows beyond one tracker, split by geography with overlapping boundary strips (20 nm), and let the owner of a track be whichever partition contains its predicted position, with explicit transfer like a sector handoff.

Conflict detection correctness. For each pair, project both tracks forward along current velocity (and turn rate if turning) at 5 s steps over 120 s, and compute closest point of approach. Flag if lateral < minimum (5 nm en route, 3 nm terminal) and vertical < 1,000 ft at the same time. Inflate the minima by position uncertainty from the filter covariance (e.g. +3σ), so an aircraft with poor radar-only coverage triggers earlier. Candidate pairs come from the grid: each detector partition owns cells plus a halo one cell wide, so a pair straddling a boundary is tested by both and deduplicated by pair key; this is cheaper and safer than trying to be clever about ownership. Cell size must be at least closing speed × look-ahead divided by the number of neighbour rings checked: two jets closing at 1,000 kt cover 33 nm in 2 min, so with 10 nm cells check 2 rings, or use 20 nm cells and 1 ring. Use cleared altitude from the flight plan to stop flagging aircraft that are levelling off 1,000 ft apart. Test with a regression library of recorded near-misses and known false alarms; a change ships only if it misses none of the near-misses.

No single point of failure and degraded mode. Every component is at least duplicated on separate hardware, networks and power. Trackers run hot-hot: both compute the same frames, displays take the primary's and switch to the standby if no frame arrives within 1.5 s, with no state transfer needed because both read the same log. The log uses three replicas across two equipment rooms. Conflict detection partitions run in pairs. Each track carries last_update; the display shows it normally under 1 sweep old, marks it "coast" (predicted, dimmed) up to ~12 s, then drops it with a visual cue. Each sensor's health shows on a status panel, and if a whole sector loses coverage the display says so instead of quietly showing fewer aircraft. The ultimate fallback is the raw-plot bypass, which shares nothing with the tracker path.

Historical tracks, audit and replay. Hot history goes to Cassandra partitioned by (flight_id, day) for fast "show me this flight", with a 45-day TTL; the log segments themselves go to WORM object storage hourly. Each segment records the SHA-256 of its content plus the previous segment's hash, and the chain head is signed and copied to a separate system daily, so deleting or editing a record is detectable. Replay runs the same tracker and detector versions (recorded with the segment) over the archived log, producing the same frames and alerts the controller saw, and overlays recorded voice timestamps for investigators.

Alert fatigue, deduplication and sector handoff. One open alert per pair key; repeated detections update last_seen instead of creating new alerts. Hysteresis: raise when predicted separation is below the minimum for 2 of the last 3 cycles; clear only after it is above minimum + 20% for 3 cycles, so alerts don't flicker. Suppress known-safe cases (parallel approaches to the same airport, formation flights with a flag, aircraft on the ground). Send each alert to the sectors owning either aircraft only, not to everyone in range. Handoff is a two-step state change: the sending sector initiates, the receiving controller accepts, and ownership changes in one transaction; until accept, both see the track highlighted and both get its alerts. Across facilities the same protocol runs over an inter-facility link, with boundary strips shared so each facility tracks aircraft before they arrive.

Trade-offs

• Hot-hot redundant trackers double hardware cost but give sub-2 s failover with no state transfer; warm standby is cheaper but blanks displays while it rebuilds.
• Deterministic processing from an ordered log makes replay and failover simple, but forbids wall-clock or random inputs in the tracker and means a bad report can crash both copies; we guard with input validation and diverse versions on the standby.
• Uncertainty-inflated minima catch more real conflicts but raise false alerts where coverage is poor; hysteresis and suppression rules claw some back.
• A 200 ms reorder window adds latency to every track to get correct physics; shorter windows drop more late reports.
• Spatial halos test boundary pairs twice; the duplicate compute is cheap compared with a missed pair.
• Keeping all of this inside one facility avoids WAN dependence but means multi-facility coordination has to be built as an explicit protocol.

Interviewer expectations

Mid-level (L4)

• Ingest reports through a queue, keep current positions in memory, push to displays over a stream.
• Check pairs for separation with a spatial index instead of all-pairs.
• Names redundancy as a requirement and adds replicas.

Senior (L5)

• Gives a latency budget per stage under 1 s and partitions by sensor and space for isolation.
• Predicts conflicts with CPA over a look-ahead, handles boundary pairs with halos, and uses hysteresis and dedup to limit alert noise.
• Designs explicit staleness states on the display and a failover path that doesn't blank screens.
• Separates hot history from immutable archive with retention numbers.

Staff (L6+)

• Makes the ordered log the source of truth so failover, recovery and replay all follow from determinism, and names the common-mode failure that creates.
• Treats measurement time, out-of-order reports and covariance-based uncertainty as correctness issues, with a regression library gating detector changes.
• Designs degraded modes down to a raw-plot bypass, tamper-evident audit, and a handoff protocol that extends to multiple facilities.