Design a metrics monitoring system

Collect metrics from every host and service, store them as time series, power dashboards over months of data, and alert when thresholds are crossed.

We are building an internal metrics platform in the style of Prometheus plus a long-term store: agents on every host and service push numeric time series, the platform keeps them for 13 months, serves dashboard queries over any range, and evaluates alert rules that page on-call.

Functional requirements

• Ingest samples (metric name, labels, timestamp, float value) from every host, container and service, at 10 s resolution by default.
• Query by metric and label matchers with aggregation (sum by (service), rate, p99 from histograms) over any time range, for dashboards and ad-hoc exploration.
• Alert rules: an expression, a threshold, a for duration (e.g. "error rate > 2% for 5 min"), routed by labels to PagerDuty, Slack or email, with silences and grouping.
• Retention: raw 10 s data for 15 days, 1 min rollups for 90 days, 1 h rollups for 13 months.
• Per-team quotas on active series, and visibility into who owns which series.

Non-functional requirements

• Write path never blocks the producer: a slow store must not stall services. Accept up to a few minutes of lag, never silent loss.
• Alerting is more critical than dashboards: an alert fires within 1 minute of the condition and keeps working when the query tier is overloaded.
• Dashboard queries: p99 under 1 s for a 24 h panel, under 3 s for a 30-day panel.
• Availability 99.9% for writes and alerts; the monitoring system must be up when everything else is down.
• Storage cost matters more than latency for data older than a week.

Back-of-the-envelope estimation

• 100k hosts × ~1,000 series each (node, container, app metrics) ≈ 100M active series.
• One sample per series per 10 s → 10M samples/s ingest.
• Raw wire size ~16 B/sample (8 B timestamp + 8 B value) before labels; with Gorilla-style compression (delta-of-delta timestamps, XOR floats) ≈ 1.4 B/sample.
• Raw storage per day: 10M × 86,400 × 1.4 B ≈ 1.2 TB/day; 15 days ≈ 18 TB, ×3 replicas ≈ 55 TB on SSD.
• 1 min rollups (5 aggregates: min, max, sum, count, last) over 90 days: 100M × 1,440 × 90 × ~6 B ≈ 78 TB in object storage. 1 h rollups for 13 months ≈ 6 TB. Object storage is cheap enough to keep both.
• Series index: 100M series × ~200 B of labels ≈ 20 GB, which fits in memory across the ingest fleet.
• Queries: ~5k dashboards open, 20 panels each, refresh every 30 s ≈ 3k queries/s, plus ~200k alert rules evaluated every 30 s ≈ 7k evals/s.

Write path (agents and services, batched, protobuf + snappy):

POST /api/v1/write
{ tenant: "payments",
  series: [ { labels: {__name__: "http_requests_total", service: "checkout", status: "500", region: "us-east-1"},
              samples: [ {ts: 1760270400000, value: 18234} ] } ] }
→ 204 | 429 {reason: "series_limit_exceeded", limit: 2000000} | 503 (retry with backoff)

Read path (dashboards, alert evaluator):

GET /api/v1/query_range?query=sum by (status)(rate(http_requests_total{service="checkout"}[5m]))
    &start=...&end=...&step=60s
→ { result: [ { labels: {status: "500"}, points: [[ts, value], ...] } ], stats: {series_scanned, bytes_read, tier: "1m"} }

GET /api/v1/series?match={service="checkout"}&start=...&end=...   → label sets
GET /api/v1/labels/{name}/values                                   → autocomplete

Alerting config:

PUT /api/v1/rules/{namespace}/{group}
{ interval: "30s",
  rules: [ { alert: "CheckoutErrorRate", expr: "sum(rate(...{status=~\"5..\"}[5m])) / sum(rate(...[5m])) > 0.02",
             for: "5m", labels: {severity: "page", team: "payments"} } ] }

POST /api/v1/silences  { matchers: {alertname: "CheckoutErrorRate"}, starts_at, ends_at, created_by, comment }

Series identity. A series is the sorted label set; series_id = hash64(sorted labels). Everything else keys off this id.

Inverted index (per 2 h block, in the TSDB, plus a global copy for routing):

Chunks. Samples for one series are appended into a chunk of ~120 samples (20 min at 10 s), compressed with delta-of-delta timestamps and XOR-encoded floats. Chunks live in memory on the ingester, are written to a local write-ahead log, and every 2 h are flushed as an immutable block (tenant, block_start) → {index, chunks} to object storage (S3/GCS).

Rollup blocks. Same block layout, but each point holds min, max, sum, count, last for a 1 min or 1 h window, so avg, rate and max stay correct after downsampling.

Metadata store (Postgres): alert rules, silences, notification routes, tenant quotas, series ownership (metric prefix → team). Small, relational, needs transactions.

Alert state (Postgres or etcd): (rule_id, label_fingerprint) → {state: pending|firing|resolved, active_since, last_eval, last_notified}.

Why these stores: time series data is append-only, read in ranges, and huge, so a purpose-built TSDB with columnar compressed chunks on object storage beats a general key-value store by 10x on cost. Cassandra or Bigtable keyed by (series_id, day) also works and is a fine answer; it trades worse compression for simpler operations.

flowchart LR
  A[Host and service agents] --> D[Distributors]
  D --> K[Kafka ingest log]
  K --> I[Ingesters with WAL]
  I --> O[(Object storage blocks)]
  K --> RU[Rollup workers]
  O --> CP[Compactor]
  CP --> O
  Q[Query frontend] --> QR[Queriers]
  QR --> I
  QR --> SG[Store gateways]
  SG --> O
  DB[Dashboards] --> Q
  R[Rule evaluators] --> QR
  R --> AM[Alert manager]
  AM --> N["Pager, Slack, email"]
  RU --> O

1. Write. An agent on each host scrapes local exporters every 10 s, batches ~500 series per request and posts to a distributor. The distributor checks the tenant's quota and series limit, rejects new series over the limit with 429, and writes the batch to Kafka partitioned by hash(tenant, series_id).
2. Ingest. Each ingester owns a set of partitions, appends samples to in-memory chunks and a local WAL, and commits Kafka offsets only after the WAL fsync. Every 2 h it cuts an immutable block and uploads it to object storage.
3. Compact and roll up. The compactor merges 2 h blocks into 1-day and 2-week blocks, deduplicates replicas, and writes 1 min and 1 h downsampled blocks. Raw blocks older than 15 days are deleted.
4. Read. The query frontend splits a long query by day, caches results per day-slice, and picks the resolution tier from the range. Queriers fan out to ingesters for the last 2 h and to store gateways for older blocks, then merge.
5. Alert. Rule evaluators, sharded by rule group, run each expression every 30 s against the queriers, track pending/firing state, and send firing alerts to the alert manager, which deduplicates, groups, applies silences and notifies.

Cardinality explosion and guardrails. Cost scales with active series, not samples. One engineer adding user_id or request_id as a label turns one series into millions and can OOM every ingester holding that tenant. Guardrails, in order of where they bite:

• Client library: refuse labels on a denylist (user_id, trace_id, url with path params) and cap values per label at 1,000 per process.
• Distributor: per-tenant active-series limit (e.g. 2M) and per-metric limit (e.g. 100k). New series over the limit are dropped with a 429 naming the metric; existing series keep flowing, so dashboards don't go blank.
• Ingester: a global per-instance ceiling (e.g. 5M series) as the last line of defence.
• Visibility: a cardinality explorer computed from the index (top metrics by series, top labels by distinct values, growth over 24 h) with an owner per metric prefix. When a limit trips, the 429 and an auto-filed ticket go to the owning team, not the platform team.
• Relabeling rules at the distributor can drop a bad label from a metric without a redeploy of the service.

Rollups, query routing and read scaling. A 30-day panel at 10 s resolution is 259k points per series; nobody can see that many pixels. The frontend picks the tier from range / step:

Rollups store sum and count (not just avg) so that averages of averages stay correct, plus max so spikes stay visible. Counters are rolled up as the last value plus a reset-adjusted increase, so rate() still works. Read scaling: the frontend splits queries into day-aligned slices, caches completed days in memcached (history is immutable, so hit rates above 80% for dashboards), and runs slices in parallel. Store gateways keep block indexes and hot chunks in local SSD cache so old data doesn't pay object-store latency (~50 ms per GET) on every request.

Alert evaluation, state and notifications. Evaluators are sharded by rule group with consistent hashing; each group runs on two replicas, and the alert manager deduplicates their output by fingerprint. Each eval writes state: a condition true for less than for is pending; at for it becomes firing and is sent; when false it becomes resolved after a short keep-firing window (e.g. 2 min) to stop flapping. State is checkpointed so a restarted evaluator doesn't re-page. The alert manager groups by (alertname, cluster) with a 30 s group wait, so 400 pods failing produce one page, not 400; repeats every 4 h if still firing; inhibition suppresses symptom alerts when a root-cause alert is firing. Alert queries hit ingesters directly for the last few minutes, and the query frontend gives the alert pool its own concurrency budget so a dashboard stampede can't delay pages.

Ingest queues, catch-up and late samples. Kafka sits between distributors and ingesters so an ingester restart or a store slowdown becomes lag, not data loss: 10M samples/s at ~20 B on the wire ≈ 200 MB/s, and 24 h of retention ≈ 17 TB, enough to replay a full day. A restarted ingester replays its WAL, then resumes its Kafka offsets; catch-up runs at 3–5x live rate, and dashboards show a "data delayed" banner while lag is over 1 min. Late samples: the in-memory head accepts out-of-order samples within a 1 h window (stored in a separate small out-of-order chunk and merged at flush); older samples go to a backfill path that writes a separate block and lets the compactor merge it. Alerts on a series whose newest sample is stale for more than 5 min evaluate as "no data", which is itself alertable, rather than silently false.

Meta-monitoring and incident mode. The monitoring stack can't watch itself alone. A small, independent Prometheus in another region scrapes the platform's own health (ingest lag, rejected samples, eval duration, notification failures), and a dead-man's-switch alert fires continuously through the alert manager to an external service that pages if the heartbeat stops. During a big incident, query load can jump 10x as everyone opens dashboards: the frontend enforces per-user concurrency (e.g. 10 queries) and a per-query limit on series scanned (e.g. 500k), sheds long-range ad-hoc queries first, and keeps the alert pool's budget untouched.

• Pull vs push. Scraping (pull) gives free liveness detection ("target down") and lets the server control load; push suits short-lived jobs and crossing firewalls. A local agent that scrapes and then pushes gets both, at the cost of one more component on every host.
• Kafka in the write path buys replay and decoupling but adds ~100 ms of latency and a large stateful system to run. Ingesters with replicated WALs (factor 3) avoid it but turn every slow store into producer back-pressure.
• Object storage for history is ~5x cheaper than SSD but each cold read pays tens of ms; SSD caches on store gateways and a results cache hide most of that for repeat dashboards, not for one-off year-long queries.
• Rollups make long-range queries fast and cheap but lose detail: a 30 s spike is visible in max but its exact shape is gone after 15 days.
• Hard series limits protect the platform but drop a team's new metrics during a launch; soft limits with alerts first, then hard limits at 1.5x, trade some risk for fewer surprises.
• Floats only, no logs or traces. Keeping the data model to numeric samples is what makes 1.4 B/sample possible; high-cardinality per-request detail belongs in tracing, not here.

Mid-level (L4)

• Defines a series as name plus labels and a sample as timestamp plus value; estimates samples/s and storage.
• Draws agents → ingestion → time series store → query and alert services, and picks a time series store with a reason.
• Explains threshold alerts with a for duration and basic notification routing.

Senior (L5)

• Sizes active series and explains why cardinality, not sample count, is the real limit; proposes per-tenant limits and a label denylist.
• Designs downsampled tiers with sum/count/max and routes queries by range; explains compression (delta-of-delta, XOR).
• Separates alert evaluation from dashboard load, handles dedup, grouping, silences and flapping; uses a durable queue so restarts become lag, not loss.

Staff (L6+)

• Treats meta-monitoring as a requirement: an independent watcher and a dead-man's switch.
• Plans for incident mode: query fairness, shedding order, alert budget protected under 10x load.
• Turns cardinality incidents into an ownership problem (explorer, metric owners, auto-routed tickets) and reasons about hot/cold storage cost against query latency with real numbers.