Design a distributed power grid control system

Source: https://www.hack2hire.com/question-bank/questions/system-design/6ab2ff05e74ebb7457a3ef14
Company: OpenAI
Difficulty: Hard

Problem statement

Monitor and control generators, substations and loads across a power grid, reacting to failures and demand changes in seconds, safely, even when parts of the network are cut off.

Requirements

We are building a grid control platform that lets a utility shed or shift load in seconds to minutes by commanding millions of distributed energy resources (smart thermostats, home batteries, EV chargers, water heaters, plus a few hundred feeder-level assets such as substation capacitor banks and peaker generators), verifying the delivered megawatts, and staying safe when parts of the network lose connectivity.

Clarify up front: is this real-time protection (milliseconds, done by relays in the substation, out of scope) or dispatch (seconds to minutes, in scope)? Who issues events (grid operator signal vs. internal forecast)? Are devices owned by customers who can opt out? Answer assumed here: dispatch only, events come from the operator or a forecaster, customers can override.

Functional requirements

• Operators create a demand-response (DR) event: target region or feeders, target reduction in MW, start, duration, priority.
• The system picks devices, computes a per-device desired state (setpoint offset, charge/discharge kW, curtail to X%), and pushes it.
• Devices acknowledge receipt and execution; telemetry (power draw, state, override flag) streams back in batches.
• A live view shows estimated reduction vs. target per feeder; the controller adds or releases devices in rounds to hit the target.
• After the event, compute final verified reduction per device and per program against a baseline, for settlement with the grid operator and customer incentives.
• Operators can cancel or amend an event; cancellation must reach every device.

Non-functional requirements

• Fan-out: first command reaches 95% of 5M online devices within 60 s.
• Safety over availability: a device that loses contact must fall back to a safe local behavior, never stay in an extreme state indefinitely.
• Commands are idempotent and versioned; replays, duplicates and out-of-order delivery never move a device backward.
• Regional partitions are normal: each region keeps controlling its own devices if it loses the central plane.
• Auditability: every command, ack and result is traceable for regulators and settlement (7-year retention).
• Telemetry may arrive hours late; results must be restatable.

Back-of-the-envelope estimation

• 5M devices, average 1.5 kW controllable → ~7.5 GW nameplate; a typical event targets 200–500 MW from 0.5–1.5M devices.
• Telemetry: one reading per device per minute when idle, every 10 s during an event. Idle: 5M / 60 ≈ 83k readings/s. Event: 1M devices / 10 s = 100k/s on top. Batches of 6–30 readings per upload → ~10–20k uploads/s.
• Reading ≈ 60 bytes compressed → 5M × 1,440/day × 60 B ≈ 430 GB/day raw; 90-day hot ≈ 40 TB in Cassandra, then downsample to 15-min intervals in object storage.
• Fan-out: 1M commands in 60 s ≈ 17k pushes/s sustained, 50k/s burst. Persistent MQTT connections: 5M across ~100 brokers at 50k each.
• Control truth (programs, events, rounds) is small: thousands of events/year, millions of command intents per event → tens of GB/year in Postgres.

API design

Operator-facing REST, device-facing MQTT (with HTTPS fallback for devices behind awkward networks).

POST /v1/events
{ "program_id": "p_summer_ac", "regions": ["west"], "feeders": ["F-1021","F-1022"],
  "target_mw": 300, "start": "2026-07-14T17:00Z", "duration_min": 120,
  "priority": "emergency", "idempotency_key": "op-8812" }
→ 201 { "event_id": "ev_91", "status": "scheduled", "eligible_devices": 1180000, "est_capacity_mw": 410 }

PATCH /v1/events/ev_91  { "action": "cancel" | "amend", "target_mw": 250 }
GET   /v1/events/ev_91/live → { "target_mw": 300, "est_reduction_mw": 287, "ci90": [262, 310],
                                "acked": 0.97, "overrides": 0.06, "round": 3 }
GET   /v1/events/ev_91/results → per-feeder verified MW, version, finalized_at

Desired state, pushed to topic dev/{device_id}/desired (retained message):

{ "device_id": "d_77", "seq": 4182, "event_id": "ev_91", "round": 1,
  "state": { "mode": "setback", "offset_f": 3 },
  "valid_from": "17:00:00Z", "expires_at": "19:00:00Z",
  "fallback": "revert_to_schedule", "sig": "ed25519:..." }

Ack on dev/{device_id}/ack: { "seq": 4182, "status": "applied" | "rejected" | "overridden", "reason": "user_opt_out", "applied_at": "..." }.

Telemetry: POST /v1/telemetry:batch or MQTT dev/{id}/tm with { device_id, boot_id, readings: [{ts, kw, state, override}] } → { "accepted_through": ts }. Readings carry device clock time; the server records receipt time too.

Recovery contract: on reconnect a device sends { last_applied_seq, boot_id }; the server replies with the current desired state (or "none, run local schedule") and the device uploads buffered telemetry from its ring buffer (48 h). Devices never act on a seq lower than the one they applied, and never act on anything past expires_at.

Data model

PostgreSQL (control truth; transactional, small, needs constraints and joins). One primary per region, synchronous replica, logical replication of reference data from the central plane.

• programs(program_id PK, kind, max_events_per_season, max_duration_min, comfort_limits jsonb)
• events(event_id PK, program_id, region, target_mw, start_at, end_at, status, version, idempotency_key UNIQUE)
• event_rounds(event_id, round, created_at, added_devices, released_devices, reason, PK(event_id, round))
• enrollments(device_id, program_id, priority, opted_out_until, PK(device_id, program_id)), index on (feeder_id, program_id).
• command_intents(event_id, round, device_id, seq, desired jsonb, PK(event_id, device_id, round)). Written in bulk (COPY) per round; this is what we meant to do, the source for audits.

Cassandra (per-device state; high write rate, partition by device).

• device_state — partition device_id: desired_seq, desired, reported_seq, reported_status, last_seen, connected_broker, boot_id. The device shadow: desired vs. reported.
• telemetry — partition (device_id, day), clustering ts: kw, state, override, received_at. TTL 90 days. Queries are "this device for this window", which the partition serves directly.

Evidence and results (object storage + warehouse). Raw telemetry is copied to Parquet partitioned by region/date. results(event_id, device_id, version, baseline_kw, actual_kw, reduction_kwh, data_completeness, computed_at) is append-only: a restatement writes version+1, never updates in place.

High-level design

flowchart LR
  OP[Operator or forecaster] --> API[Event API]
  API --> PG[(Postgres control truth)]
  API --> PLN[Dispatch planner]
  PLN --> PG
  PLN --> FAN[Regional fan-out workers]
  FAN --> SH[(Cassandra device shadow)]
  FAN --> MQ[MQTT broker fleet]
  MQ <--> DEV[Devices and feeder assets]
  MQ --> ING[Telemetry ingest]
  ING --> K[Kafka telemetry topic]
  K --> TS[(Cassandra telemetry)]
  K --> EST[Live estimator]
  EST --> CTL[Closed loop controller]
  CTL --> PLN
  K --> LAKE[Object storage]
  LAKE --> MV[Verification batch jobs]
  MV --> RES[(Results store)]

Each region runs its own copy of planner, fan-out, brokers, ingest, estimator and controller. The central plane only holds program config and cross-region reporting.

1. Event to round 1. The API validates the event against program limits (max duration, events per season), writes it to Postgres, and the planner picks devices: eligible enrollments on the target feeders, sorted by priority and expected kW from recent telemetry, until expected reduction ≥ target × 1.15 (headroom for overrides and non-response). It writes command_intents for round 1 in one transaction and hands device lists to fan-out workers, sharded by broker.
2. Dispatch and ack. Each worker bumps desired_seq in the shadow, publishes the retained desired-state message, and waits for acks. Missing acks after 20 s get one republish; after that the device is marked unreachable and excluded from the estimate.
3. Measure, verify, correct. Telemetry flows through Kafka. The estimator compares each device's live kW with its baseline and sums per feeder every 30 s. The controller compares the sum with target and, every 2–5 minutes, opens a new round: add devices if short, release some if well over.
4. Device reconnect. A device reconnecting gets its retained desired message from the broker, compares seq with what it applied, applies if newer and unexpired, and uploads its buffered telemetry. The ingest path treats late readings like any others; the verification job picks them up.

Why it holds together: the device shadow is the only per-device truth, every message carries a monotonically increasing seq and an expiry, and the device always has a safe local behavior. So the loop survives lost, duplicated, reordered or late messages without coordination.

Deep dives

Fanning out to 5M devices in 60 seconds. Do not loop over devices from one service. The planner splits the device list by broker (devices hash to a broker by device_id), so each of ~100 fan-out workers owns 10–50k devices and talks to one broker. Each publishes at ~2k msg/s → 50k devices in 25 s. Shadow updates go to Cassandra in unlogged batches by partition at ~5k writes/s per worker. Retained messages mean a device that blinks off and on still finds its command without a second publish. For broadcast-style programs (all water heaters in a region, same instruction) use a group topic grp/{program}/{region} plus per-device topics only for exceptions: one publish reaches 500k devices. Priority lanes: emergency events preempt economic ones at the worker queue, and cancellations jump ahead of everything.

Verification against a counterfactual baseline. Reduction is "what would have been consumed" minus "what was consumed", and the first term is never observed.

Pick: a randomized control group for program-level truth and settlement disputes, plus a regression baseline per device for live estimates and customer incentives. Exclude a 1-hour pre-event window from day-of adjustment to blunt pre-cooling, and add the post-event snapback (load rebound) to the reported net.

Late telemetry and restatement. Three tiers of numbers: live (every 30 s, from whatever has arrived, with a confidence band that widens as completeness drops), provisional (T+1 h, devices with ≥ 80% of readings), final (T+72 h, or when the ingest watermark passes the event end for 99% of devices). Missing intervals are imputed from the baseline (zero credit), never from the target. When late data arrives after final, the job writes a new results version with a reason; settlement uses the version at the settlement cut-off and records later versions as adjustments. Watermarks are per device using received_at, because device clocks drift; readings with clock skew > 5 min are re-timed against the server's receipt time.

Closing the loop without oscillation or harm. Treat the fleet as a slow, noisy plant. The controller is a PI loop on the aggregate error with a deadband (±3% of target) and a rate limit (change at most 10% of the target per round), with rounds no faster than every 2 minutes because thermostats take 5–10 minutes to show the effect. Adding devices is cheap; releasing is staggered with random 0–10 minute jitter so 200k air conditioners do not restart together and create a spike larger than the reduction. The same jitter applies at event end. Hard safety limits sit below the controller and win every time: per-device comfort and battery state-of-charge floors enforced on the device, per-feeder ramp limits (MW/min) from the distribution operator, and a refusal to dispatch any feeder whose voltage telemetry is already outside band.

Partitions and bad commands. If a region loses the central plane, it keeps running its events from its own Postgres and brokers; new cross-region events wait. If a device loses its broker, the expires_at and fallback in the last command decide what happens: it reverts to its schedule at expiry, so the worst case is an event that ends early, not a home stuck at 85 °F. A bad command (wrong offset, wrong region) is contained by staged rollout: round 1 goes to 1% canary devices, the controller checks acks and kW response for 60 s, then releases the rest. A cancel is a new desired state with a higher seq and fallback now, published on the emergency lane. All commands are signed; devices reject unsigned or out-of-program states (offset > program max).

Trade-offs

• Desired-state push (shadow + retained message) over imperative commands: survives reconnects and duplicates, but you give up fine-grained step sequencing on the device.
• Regional autonomy over a single global controller: partitions don't stop control, but cross-region events can't rebalance capacity during a partition.
• 15% dispatch headroom plus rounds: hits target reliably, but asks more customers per event and uses up their seasonal event budget faster.
• Control groups give unbiased verification but hold back 5–10% of capacity on every event.
• Final results at T+72 h with restatement: accurate settlement, but customers and the grid operator wait three days for official numbers.
• Device-side fallback and expiries put safety on hardware we don't fully control; devices that lie (spoofed telemetry, firmware ignoring commands) are caught only statistically, by comparing reported response to feeder-level meter data and to the control group.
• Multiple programs targeting one device are resolved by a single arbiter in the planner (highest priority wins, one active event per device), which is simple but leaves some capacity unused.

Interviewer expectations

Mid-level (L4)

• Separates the operator API, device messaging, and telemetry pipeline.
• Uses a pub/sub protocol (MQTT) with acks and retries rather than polling.
• Sizes telemetry ingestion and picks a time-series-friendly store.

Senior (L5)

• Models commands as versioned desired state with expiry; explains reconnect and idempotency.
• Shards fan-out by broker to hit the 60 s target, with priority for cancellations.
• Explains baseline verification and why live estimates differ from final results.
• Names safe device-side fallbacks for lost connectivity.

Staff (L6+)

• Designs the closed loop as a control problem: deadband, rate limits, staggered release, snapback.
• Plans for regional partitions, canary rounds for bad commands, and dishonest devices.
• Builds restatable, versioned results for settlement and audit, and arbitrates competing programs per device.
• Draws the line between this dispatch system and millisecond protection systems it must never replace.