Design a real-time temperature monitoring system

Collect readings from millions of sensors every few seconds, show live and historical temperatures, and alert when a sensor goes out of range or silent.

We're building a monitoring service for a fleet of fixed, geolocated temperature sensors (warehouses, cold chain, data centers, outdoor stations): it ingests a reading from every sensor every few seconds, shows live and historical temperatures per sensor and as a zoomable map heatmap, and alerts when a sensor goes out of range or stops reporting.

Functional requirements

• Sensors (or their gateways) push batches of readings: sensor_id, timestamp, temperature, optional battery/RSSI.
• Query one sensor's history at any range: last 10 minutes at raw resolution, last year at hourly resolution.
• Heatmap: for a map viewport and zoom level, show the current (or a past) temperature per area, refreshing every ~10 s.
• Alert rules per sensor or group: above/below threshold for N seconds, rate of change, and silence (no reading for X seconds).
• Register sensors with location, owner (tenant), and expected reporting interval.

Out of scope: device firmware and OTA, actuation (turning chillers on), arbitrary ad-hoc analytics (export to a lake instead).

Non-functional requirements

• Ingest at sustained 1M readings/s with 2× headroom for reconnect storms; never drop an acknowledged reading.
• Freshness: a reading is visible on the live view and in alert evaluation within 5 s (p99).
• Alert latency: threshold breach to notification under 10 s; silence detection within interval + 30 s.
• History queries under 500 ms p99 for any range; heatmap tiles under 200 ms.
• Tolerate late and out-of-order data (gateways buffer for hours during outages) and correct aggregates when it arrives.
• Cost: raw data is huge and mostly unread after a week, so retention is tiered.

Back-of-the-envelope estimation

• 5M sensors × 1 reading / 5 s = 1M readings/s, ~86B readings/day.
• Wire size ~40 B/reading in a batch → 40 MB/s ingress, ~3.5 TB/day into the log (before replication).
• Raw in a TSDB with delta-of-delta timestamps and XOR floats ≈ 2 B/point → ~170 GB/day, ~1.2 TB/week hot.
• 1-minute rollups (min, max, sum, count, last ≈ 24 B): 5M × 1,440 = 7.2B rows/day ≈ 170 GB/day raw, ~40 GB compressed.
• 1-hour rollups: 120M rows/day ≈ 1 GB/day compressed; one year ≈ 400 GB. Cheap to keep forever.
• Heatmap: at zoom 12, the world is 16.7M tiles but sensors cover maybe 200k of them; live tile cache is a few GB.
• Alert state: 5M sensors × ~100 B (last value, last seen, breach start) = 500 MB, fits in memory across a stream-processing job.

Ingestion is push, over MQTT or HTTPS from gateways (sensors are constrained devices behind NAT, so pull doesn't work; the gateway batches to amortize TLS and headers).

POST /v1/ingest                     (gateway auth: mTLS client cert)
{ "gateway_id": "gw-81", "batch_seq": 99102,
  "readings": [ { "sensor_id": "s-1234", "ts": 1760270400123, "temp_c": 4.2, "battery": 87 }, ... ] }  // up to 1,000
→ 202 { "accepted": 1000, "batch_seq": 99102 }

(gateway_id, batch_seq) makes retries idempotent; the 202 is returned only after the batch is durably in the log.

GET /v1/sensors/{id}/series?from=...&to=...&step=auto
→ { "resolution": "1m", "points": [ { "t": ..., "min": 3.9, "max": 4.4, "avg": 4.1 } ] }

GET /v1/tiles/{layer}/{z}/{x}/{y}?at=now|<ts>        layer = avg_temp | max_temp
→ { "z": 12, "x": 655, "y": 1583, "as_of": ..., "cells": [ { "geohash": "9q8yyk", "avg": 18.3, "max": 22.1, "n": 41 } ] }
   Cache-Control: max-age=10 for "now", immutable for closed past buckets

POST /v1/alert-rules  { "scope": { "group_id": "freezers-dc3" }, "type": "range", "min": -25, "max": -15, "for_s": 120, "notify": ["pagerduty:svc-77"] }
POST /v1/alert-rules  { "scope": { "sensor_id": "s-1234" }, "type": "silence", "after_s": 60 }
GET  /v1/alerts?state=firing&tenant=...
WS   /v1/live?sensors=s-1,s-2   → pushes latest readings for an open dashboard

step=auto picks the coarsest resolution that still gives ≥ 300 points for the range, so a chart never pulls more than a few thousand points.

Invariants: an aggregate is a pure merge of (min, max, sum, count) so any bucket can be recomputed from finer buckets; each rollup row carries a version so a correction replaces rather than double-adds; a reading's identity is (sensor_id, ts), so duplicates collapse.

flowchart LR
  S[Sensors] --> GW[Gateways]
  GW --> ING[Ingest API]
  ING --> K[(Kafka readings)]
  K --> RAW[Raw writer]
  RAW --> TS[(TSDB raw)]
  K --> AGG[Rollup stream job]
  AGG --> RU[(Rollup tables)]
  AGG --> GEO[(Geo rollups)]
  K --> AL[Alert evaluator]
  AL --> N[Notifier]
  K --> LV[Latest writer]
  LV --> R[(Redis latest)]
  RU --> Q[Query service]
  TS --> Q
  GEO --> TILE[Tile service]
  TILE --> CDN[CDN cache]
  R --> WS[Live push]
  TS --> ARC[(S3 archive)]
  REG[(Sensor registry)] -.-> AGG
  REG -.-> AL

1. Ingestion. Gateways batch ~1,000 readings and POST them. The stateless ingest tier validates schema and auth, drops readings with timestamps far in the future, writes the batch to Kafka with acks=all, and only then returns 202. Kafka absorbs bursts; every consumer below reads independently at its own pace.
2. Raw storage. The raw writer consumes by partition, buffers ~1 s, and bulk-inserts into the TSDB shard owning those sensors. Upsert on (sensor_id, ts) makes redelivery harmless.
3. Multi-resolution aggregation. A Flink-style job keyed by sensor_id keeps 1-minute tumbling windows in state, emits each window when the watermark passes, and also merges into the open 1-hour bucket. It enriches with the sensor's geohash from a cached registry and emits per-cell partials, which a second keyed stage (by geohash cell) merges into geo_1m.
4. Alerting. A separate job keyed by sensor_id evaluates rules on every reading (range, duration, rate of change) and uses per-sensor timers for silence. State changes (ok → firing → resolved) go to the notifier, which dedupes, groups, and routes.
5. Live. The latest writer keeps Redis current; the WebSocket tier fans out updates for sensors a dashboard is watching.
6. History and heatmap reads. The query service picks a resolution from the range and reads one table. The tile service merges geo cells under a tile, caches the result, and the CDN serves repeated viewers.

Multi-resolution rollups. Charts over a year at raw resolution would scan 6.3M points per sensor; at 1 h it is 8,760. Keep a ladder and choose by range:

Store min, max, sum, count, last, never avg: averages don't merge, sums and counts do, and min/max are what an operator actually looks for in a cold-chain excursion that an average would hide. 1 h is built from 1 m buckets (streaming, or a batch job at hour close), 1 d from 1 h. Percentiles, if needed, use a mergeable sketch (t-digest) in a separate column rather than raw rescans.

Late data and corrections. Gateways offline for 6 hours will flush 4,320 readings per sensor at once. Watermark the rollup job at event time with 2 minutes of allowed lateness so normal jitter lands in the right window. Anything later goes to a late-readings topic; a correction worker groups it by (sensor_id, bucket), re-reads the raw rows for that bucket from the TSDB, recomputes the aggregate, and writes it with version + 1, then marks the parent 1 h/1 d buckets and affected geo cells dirty for recompute. This keeps the streaming path simple and bounded while history ends up exact. Tiles for corrected past buckets are purged from the CDN by key. Late readings are still sent to the alert job, but rules ignore readings older than the rule window so a stale backfill doesn't page someone about a problem that ended hours ago (it's logged as a historical excursion instead, which cold-chain compliance needs).

Heatmap tile serving. Tiles use the standard web-map z/x/y scheme. The base layer is geohash precision-6 cells (~1.2 × 0.6 km) aggregated per minute. For a tile at zoom ≥ 11, return cells as-is; for lower zooms, merge cells up to the precision that gives ~64×64 cells per tile (precision 5 at zoom 8, 4 at zoom 5). Because aggregates are mergeable, one base table serves every zoom. Precompute and store tiles for zooms 0–8 every minute (few tiles, heavy merges); compute zoom 9+ on demand and cache for 10 s in Redis keyed tile:{layer}:{z}:{x}:{y}:{bucket}. Many dashboards watching the same region at "now" all hit the CDN with a 10 s TTL, so origin load scales with distinct tiles, not viewers. Past buckets are immutable once the lateness window closes, so they get long cache lifetimes. Sparse regions: return n per cell so the client can fade cells backed by one sensor and not interpolate across empty space.

Ingestion resilience. The ingest tier is stateless behind a load balancer, autoscaled on CPU and Kafka produce latency. Backpressure: if Kafka produce latency exceeds 500 ms, return 503 with Retry-After and let gateways buffer locally (they already hold hours of data for network outages); never accept what you can't persist. Retry storms after a regional outage are smoothed by jittered backoff in the gateway client and a per-gateway token bucket. Kafka: 512 partitions, RF 3, min.insync.replicas=2, so one broker loss loses nothing. Consumers are idempotent (upsert raw, versioned rollups), so at-least-once delivery is enough and exactly-once transactions are only needed in the rollup job's checkpoint-to-sink path. Each consumer group is isolated: a slow TSDB backs up only the raw writer's lag, not alerts. Alert evaluation is the one consumer with a lag SLO (< 5 s) and its own capacity.

Silence detection at 5M sensors. Scanning every sensor's last-seen every few seconds is 5M reads per sweep. Instead, the alert job registers a processing-time timer per sensor at last_seen + after_s, re-armed on each reading; timers live in the job's keyed state (RocksDB) and fire only for sensors that actually went quiet. To avoid a page storm when a whole gateway or site drops, the notifier groups silence alerts by gateway: if > 50% of a gateway's sensors go silent within a minute, emit one "gateway offline" alert and suppress the individual ones. A silence timer is also suppressed while the ingestion pipeline itself is lagging, so a Kafka backlog doesn't look like 5M dead sensors.

• Streaming rollups vs. query-time aggregation. Precomputing makes every chart and tile cheap and predictable but costs a stream job, extra storage (~25% of raw), and correction machinery for late data. Query-time aggregation is simpler but long-range charts would scan millions of points.
• Bounded lateness + correction path vs. long allowed lateness. A 2-minute watermark keeps state small and live data fresh; recent buckets may be revised later. Holding windows open for hours would avoid corrections but multiplies state size and delays every rollup.
• Purpose-built TSDB vs. wide-column store. A TSDB gets ~10× compression and fast range scans; Cassandra/DynamoDB scales writes trivially but stores each point at 30–50 B and makes rollups your problem.
• Push from gateways vs. pull. Push fits devices behind NAT and lets gateways batch; it gives up server-controlled pacing, so backpressure has to be explicit (503 + local buffering).
• Geohash cells vs. exact sensor points on the map. Cells make tiles cacheable and mergeable at every zoom but lose exact placement at low zoom; individual sensors are shown only past zoom 14.
• Tiered retention. Raw for 7 days hot, then Parquet on S3; rollups kept for years. A request for raw data from months ago becomes a slow archive query instead of an instant one.

Mid-level (L4)

• Clear requirements and estimates that land near 1M writes/s and TB/day.
• Push ingestion through a queue into a time-series store, with a query API and a simple threshold alert.
• Recognizes long-range charts need downsampled data.

Senior (L5)

• Mergeable rollups (min/max/sum/count) at multiple resolutions and picking resolution by range.
• Idempotent ingestion, Kafka partitioned by sensor, independent consumers so alerts don't wait on storage.
• Event-time windows with watermarks, a plan for late data, and timer-based silence detection.
• Tile scheme with caching for live and past views.

Staff (L6+)

• Correction path with versioned buckets, dirty-parent recompute and CDN purge, and how late backfills interact with alerting and compliance.
• Failure modes: reconnect storms, backpressure end to end, gateway-level alert grouping, suppressing silence alerts during pipeline lag.
• Cost model across tiers (hot raw, rollups, archive) and operational metrics: consumer lag, sensor fleet health (battery, dropout rate), freshness SLO per stage.