Design a logging service

Ingest logs from every service at terabytes per day, make them searchable within seconds, and keep them cheaply for months.

We are building a multi-tenant, centralized logging service: every internal service ships structured logs through an SDK, the service makes them full-text searchable within seconds, and it keeps them cheaply for months under per-tenant retention rules.

Clarify first: who are the tenants (teams or external customers)? Is losing a log line ever acceptable (debug logs yes, audit logs no)? How fresh must search be, and how far back do people actually query? Answers here: tenants are internal teams, at-least-once delivery, search within 5 s, and 95% of queries touch the last 24 hours.

Functional requirements

• Ingest batches of structured log events (timestamp, service, level, message, trace id, free-form attributes) from SDKs and agents.
• Search by time range plus filters (service, level, host, trace id, attributes) and full-text terms over the message; sort newest first; page through results.
• Fetch a single event by id with surrounding context (the 50 lines before and after from the same host).
• Per-tenant retention policy: hot days (fast search) and total days (cold, slower search), changeable without a deploy.
• Tail mode: live stream of matching events for a running query.

Non-functional requirements

• Ingest-to-searchable p99 under 5 s; ingest acknowledgement p99 under 100 ms.
• No acknowledged event is lost (durable once the API returns 202).
• Search over the last hour returns in under 1 s p95; a 30-day cold query may take tens of seconds.
• Tenant isolation: one noisy tenant cannot slow another's ingest or queries.
• Cost: storage per GB-month dominates, so cold data must sit in object storage, not on search-cluster SSDs.
• Logging must never take down the caller: the SDK drops before it blocks.

Back-of-the-envelope estimation

• 10 TB/day raw across ~2,000 services ≈ 115 MB/s average, ~350 MB/s at a 3× peak.
• Average event 500 B → ~230k events/s average, ~700k events/s peak.
• Hot tier, 7 days in Elasticsearch: 70 TB raw; with ~1.1× index overhead after best_compression and one replica ≈ 150 TB of SSD. At ~4 TB usable per data node, ~40 hot nodes.
• Cold tier, 90 days in object storage as Parquet with zstd (~8–10× compression): ~1.1 TB/day × 90 ≈ 100 TB, about $2.3k/month on S3 standard versus roughly 15× that on cluster SSD.
• Kafka buffer, 24 h retention, replication factor 3: 10 TB × 3 ≈ 30 TB across the brokers.
• Queries: ~200 QPS at peak, mostly dashboards and on-call engineers.

POST /v1/logs:batch                       (gzip, max 5 MB or 10k events)
  headers: Authorization: Bearer <ingest key>, Idempotency-Key: <batch uuid>
  body: { "resource": {"service": "checkout", "host": "ip-10-1-2-3", "env": "prod"},
          "events": [{ "ts": "2026-10-12T08:01:02.123Z", "level": "ERROR",
                       "msg": "payment declined", "trace_id": "ab12…",
                       "attrs": {"order_id": "o_991", "status": 402} }] }
  202 { "accepted": 9998, "rejected": [{"index": 17, "reason": "ts_out_of_range"}] }
  429 { "retry_after_ms": 2000 }      503 { "retry_after_ms": 500 }

POST /v1/logs:search
  body: { "query": "level:ERROR AND service:checkout AND \"declined\"",
          "from": "2026-10-12T07:00Z", "to": "2026-10-12T08:00Z",
          "limit": 100, "cursor": null, "tier": "auto" }
  200 { "events": [...], "next_cursor": "eyJ0cyI6…", "tiers_scanned": ["hot"],
        "partial": false, "took_ms": 340 }
  (cold ranges return 202 { "query_id": "q_77" }; poll GET /v1/queries/q_77)

GET /v1/logs/{event_id}?context=50        200 { "event": {...}, "before": [...], "after": [...] }

PUT /v1/tenants/{tenant_id}/retention
  body: { "hot_days": 7, "total_days": 90, "rules": [{"match": "level:DEBUG", "total_days": 3}] }
  200 { "effective_at": "2026-10-13T00:00Z" }

Recovery contract. The server returns 202 only after the batch is committed to Kafka with acks=all. Every batch carries an Idempotency-Key, and each event gets a deterministic id hash(tenant, batch_key, index). That id is the Elasticsearch _id, so a client retry after a timeout overwrites instead of duplicating. 429 and 503 mean "nothing was stored, retry after the hint". A partial 202 lists rejected indexes, which the client must not retry.

Elasticsearch hot index, one data stream per tenant tier, rolled over at 50 GB per primary shard or 1 day:

_source is kept with best_compression. Shared indexes route by tenant_id so a tenant's documents sit on few shards; tenants over 200 GB/day get their own data stream.

Cold tier, object storage: s3://logs-cold/tenant=<id>/date=<yyyy-mm-dd>/hour=<hh>/part-<n>.parquet, files of 256–512 MB, sorted by (service, timestamp). Each file has a sidecar with per-column min/max, a bloom filter on trace_id, and a small token bloom filter on msg terms, so a cold scan skips most files.

Tenant metadata, Postgres (small, relational, transactional):

• tenants(id PK, name, plan, ingest_quota_mb_s, query_quota_qps, dedicated_index bool)
• retention_policies(tenant_id FK, match_expr, hot_days, total_days, updated_at)
• ingest_keys(key_hash PK, tenant_id, scopes, revoked_at)
• promoted_fields(tenant_id, field, type)

Gateways and indexers cache this with a 30 s refresh.

Access patterns. (1) Time-bounded filter plus text on the last hours: hot index with index sorting and the time range pruning shards. (2) Trace id lookup: term query over hot, then bloom-filtered cold. (3) Context around one event: host plus a time range of ±N lines. (4) Rare month-long audits: async cold scan.

Storage trade-offs. ES gives sub-second search but costs about $0.10 per GB-month of replicated SSD and heap. Parquet on S3 costs about $0.023 and has no replica tax, but every query is a scan. An inverted index on everything forever is the failure mode to avoid.

flowchart LR
  SDK[Service SDK] --> LB[Load balancer]
  AG[Host agent] --> LB
  LB --> GW[Ingest gateway]
  GW --> MD[(Tenant metadata)]
  GW --> K[(Kafka logs topic)]
  K --> IX[Indexer pool]
  K --> AR[Archiver]
  IX --> ES[(Elasticsearch hot)]
  AR --> S3[(Object storage cold)]
  U[Engineer UI] --> QS[Query service]
  QS --> ES
  QS --> CQ[Cold query workers]
  CQ --> S3
  LC[Lifecycle manager] --> ES
  LC --> S3
  LC --> MD

1. Ingest. The SDK buffers events in memory, batches every 1 s or 1 MB, gzips, and POSTs. The gateway authenticates the key, checks the tenant's token bucket, validates the schema and timestamps, then produces to Kafka keyed by tenant_id:service (256 partitions). It returns 202 after acks=all.
2. Indexing. Indexer consumers read Kafka and redact secrets (card numbers, tokens) with tenant regex rules. They send _bulk requests of 5–10 MB to ES and commit offsets only after the bulk succeeds. The ES refresh_interval of 1–2 s makes events searchable in about 3 s end to end.
3. Archiving. A separate consumer group buffers per (tenant, hour), writes Parquet files to S3 every 5 minutes or 256 MB, and compacts small files hourly. Cold data does not depend on ES, so an ES outage loses no history.
4. Query. The query service parses the query, injects tenant_id as a mandatory filter, and splits the time range. Ranges inside hot retention go to ES. Older ranges go to cold workers that prune by partition path and sidecar stats, then scan Parquet in parallel. Results merge by timestamp with an opaque cursor (ts, event_id).
5. Tiering. The lifecycle manager runs ILM (index lifecycle management) daily: rollover, force-merge to 1 segment after a day, and delete hot indexes past hot_days. It also deletes S3 prefixes past total_days, or sets object lifecycle rules per tenant prefix.

SDK buffer contract. The SDK is the first line of durability and the last line of protection for the caller. It keeps a bounded ring buffer (default 16 MB or 50k events) and a background flusher. A log call never blocks the request thread.

Services that cannot lose a line (audit, billing) write to a local file instead, and the host agent tails it with a persisted offset. That trades latency for durability across process crashes.

Ingest pipeline scaling. At 700k events/s peak, gateways are stateless; at ~20k events/s per core, ~40 pods with headroom. Kafka absorbs the gap between ingest and indexing: if ES slows, consumer lag grows instead of the gateway rejecting traffic, and 24 h of retention covers a full-day ES outage. Indexers scale on consumer lag, and each pod sustains ~25 MB/s of bulk indexing. ES write throughput is the real bottleneck, so the design sizes shards at 30–50 GB, uses one replica, sets refresh_interval to 2 s rather than 1 s (about 30% more indexing throughput), and lets ES auto-generate IDs only for tenants that opt out of dedup. Hot partitions come from one huge service: key it by tenant:service:hash(host) % 8 to spread it.

Tenant-aware rate limiting and noisy neighbors. Each tenant has an ingest token bucket in MB/s (burst = 60 s of quota), enforced at the gateway with Redis plus a local share per pod. Over-quota traffic first gets 429. Sustained overload then routes to a separate logs-overflow topic, which indexers drain at low priority, so the main topic's lag stays near zero for everyone else. On the query side, per-tenant concurrency limits (e.g. 10 in-flight searches), a 30 s timeout, a cap of 10k shard-docs per aggregation bucket, and ES search thread-pool isolation through dedicated indexes for the largest tenants stop one regex-over-30-days query from starving others.

Retention lifecycle and hot-to-cold tiering. Retention is evaluated per tenant and per rule. A DEBUG rule of 3 days is applied at write time by tagging the event with a ttl_class, which routes it to a short-lived index and S3 prefix. Delete then means dropping a whole index or prefix, never running delete_by_query. Changing hot_days from 7 to 14 is cheap going forward but cannot restore data already dropped from ES. Re-hydration from Parquet into a temporary index is an explicit, quota'd job. Compliance deletes (GDPR) on immutable Parquet need a tombstone list applied at query time plus a monthly rewrite of affected files.

Query performance under high-cardinality fields. Request ids, user ids and URLs explode the ES mapping and terms dictionaries when indexed as top-level keywords. The flattened field indexes all attributes as one keyword field, so the cluster mapping stays bounded (limit 1,000 fields per index). Tenants can promote up to 20 fields to typed columns when they need range queries or aggregations. High-cardinality aggregations (top 10 user ids over a day) use terms with execution_hint: map on a sampled subset, or are pushed to the cold engine. Index sorting on @timestamp desc lets "latest 100 errors" stop after the first segment hits.

Adaptive sampling under sustained overload. When a tenant stays over quota for 5 minutes, or the indexers' lag passes 2 minutes globally, the gateway switches that tenant to sampling. It keeps 100% of WARN+, all events with a trace id flagged as sampled, and 1-in-N of INFO/DEBUG, with N chosen so throughput meets the quota. Each kept event carries sample_rate: N so counts in the UI are multiplied back up. The archiver still writes everything unsampled to S3 when storage budget allows, so the full data can be re-hydrated later.

• Elasticsearch for the hot tier gives fast full-text search but is the expensive, operationally heavy part. A label-only index (Loki style) costs ~10× less but makes free-text search a brute-force scan.
• Kafka between gateway and indexers adds ~1 s of latency and a cluster to run, but buys replay, backpressure absorption and independent consumers (archive, alerting).
• Writing cold data straight from Kafka, not by exporting from ES, doubles consumers but keeps the cold tier complete during ES incidents.
• At-least-once with deterministic ids gives dedup in ES only. Cold Parquet may hold duplicates, and the query layer dedups by event_id on merge.
• Shared indexes routed by tenant keep the shard count low but weaken isolation. Dedicated indexes for big tenants cost more shards and heap.
• SDK drop-on-full protects services at the price of silent loss, which the dropped_events metric makes visible but does not prevent.
• Sampling under overload keeps the platform up but makes rare INFO-level events unsearchable until re-hydrated.

Mid-level (L4)

• Clear ingest → queue → index → search pipeline with batching and an inverted index for text search.
• Reasonable estimates for daily volume, events per second and storage.
• Knows logs need time-based indexes and a retention delete.

Senior (L5)

• A precise ack and retry contract with idempotent event ids, and a bounded SDK buffer that never blocks the caller.
• Hot/cold tiering with cost numbers, ILM, and whole-index deletes instead of row deletes.
• Per-tenant quotas, mapping explosion prevention, and a Kafka buffer that absorbs indexing slowdowns.

Staff (L6+)

• Isolation as a design axis: overflow topics, dedicated indexes for big tenants, query concurrency limits, and adaptive sampling with re-weighted counts.
• Treats ES as a rebuildable cache over Kafka plus object storage, and plans re-hydration and compliance deletes on immutable files.
• Ties choices to cost per GB-month and to the 95%-last-24h query distribution, and says what to measure (consumer lag, drop counters, ingest-to-searchable p99) before scaling.