Alert routing service with silences and escalation

This is a practical round: a small service with a spec that grows in three parts. The grade is mostly about whether the code stays easy to change when part 3 arrives, so spend two minutes on the data model before typing.

What an alert is. The monitoring system calls fire(labels, now) on every evaluation cycle (say every 30 seconds) for as long as the condition holds, then resolve(labels, now) once. So fire arrives many times for the same alert, and the router's main job is to turn that stream into a few notifications. Identity is the label set: {"alertname": "DiskFull", "host": "db1", ...}. Sort the labels into a string (the fingerprint) so the order the caller used does not matter.

Part 1: routing. Routes are an ordered list. Walk it; the first route whose matchers all equal the alert's labels wins, unless that route says continue, in which case keep walking and collect more receivers. No match: the default receiver. This is how Alertmanager-style routing trees behave, flattened to a list, and it is the same "first match unless fall-through" rule as a firewall.

Part 2: dedupe, repeat, resolve. State per fingerprint: when each receiver was last notified. A fire notifies a receiver only if it never heard about this alert, or heard more than repeat_interval ago. resolve removes the state and sends resolved only to receivers that were actually notified. Telling someone an alert they never saw is over is noise, and in a pager it is worse than noise.

Part 3: silences, ack, escalation.

• A silence is (matchers, start, end). While active (start <= now < end, half-open so back-to-back silences do not overlap or leave a gap), matching alerts are tracked but not sent. The state must still be tracked, so a resolve during the silence cleans up.
• An unacked alert on a route with escalate: {after, to} pages to once, after seconds after the first notification. Measure from the first page, not the first fire: if the alert was silenced for an hour, nobody has been paged yet and escalating immediately would wake a manager about something no one was told about.
• tick(now) runs escalation. Keeping time-driven work in tick (instead of a timer per alert) keeps the whole thing deterministic and testable with a fake clock.

Data model.

from __future__ import annotations

from dataclasses import dataclass, field


def fingerprint(labels: dict) -> str:
    """Stable identity of an alert: its labels, sorted."""
    return ",".join(f"{k}={labels[k]}" for k in sorted(labels))


def matches(matchers: dict, labels: dict) -> bool:
    return all(labels.get(k) == v for k, v in matchers.items())


@dataclass
class Alert:
    labels: dict
    notified_at: int | None = None              # first notification time
    last_sent: dict = field(default_factory=dict)  # receiver -> time
    acked: bool = False
    escalated: set = field(default_factory=set)    # escalation receivers paged


class AlertRouter:
    def __init__(self, routes: list, default_receiver: str, repeat_interval: int) -> None:
        self.routes = routes
        self.default_receiver = default_receiver
        self.repeat_interval = repeat_interval
        self._alerts: dict = {}    # fingerprint -> Alert, in first-fire order
        self._silences: dict = {}  # id -> (matchers, start, end)
        self._next_silence = 1

    # -- part 1: routing -----------------------------------------------------
    def matching_routes(self, labels: dict) -> list:
        out = []
        for route in self.routes:
            if matches(route["match"], labels):
                out.append(route)
                if not route.get("continue", False):
                    break
        return out

    def route(self, labels: dict) -> list:
        receivers = []
        for r in self.matching_routes(labels):
            if r["receiver"] not in receivers:
                receivers.append(r["receiver"])
        return receivers or [self.default_receiver]

    # -- part 2: dedupe, repeat, resolve -------------------------------------
    def fire(self, labels: dict, now: int) -> list:
        fp = fingerprint(labels)
        alert = self._alerts.get(fp)
        if alert is None:
            alert = self._alerts[fp] = Alert(dict(labels))
        if self._silenced(labels, now):
            return []
        sent = []
        for receiver in self.route(labels):
            last = alert.last_sent.get(receiver)
            if last is None or now - last >= self.repeat_interval:
                alert.last_sent[receiver] = now
                sent.append((receiver, "firing", fp))
        if sent and alert.notified_at is None:
            alert.notified_at = now
        return sent

    def resolve(self, labels: dict, now: int) -> list:
        fp = fingerprint(labels)
        alert = self._alerts.pop(fp, None)
        if alert is None:
            return []
        # Only people who heard it was firing hear it is over.
        paged = list(alert.last_sent) + sorted(alert.escalated)
        return [(receiver, "resolved", fp) for receiver in paged]

    # -- part 3: silences, ack, escalation -----------------------------------
    def silence(self, matchers: dict, start: int, end: int) -> int:
        if not matchers:
            raise ValueError("a silence needs at least one matcher")
        if end <= start:
            raise ValueError("a silence must end after it starts")
        sid = self._next_silence
        self._next_silence += 1
        self._silences[sid] = (dict(matchers), start, end)
        return sid

    def expire_silence(self, silence_id: int) -> bool:
        return self._silences.pop(silence_id, None) is not None

    def ack(self, labels: dict) -> bool:
        alert = self._alerts.get(fingerprint(labels))
        if alert is None:
            return False
        alert.acked = True
        return True

    def tick(self, now: int) -> list:
        sent = []
        for fp, alert in self._alerts.items():
            if alert.acked or alert.notified_at is None or self._silenced(alert.labels, now):
                continue
            for r in self.matching_routes(alert.labels):
                esc = r.get("escalate")
                if not esc or esc["to"] in alert.escalated:
                    continue
                if now - alert.notified_at >= esc["after"]:
                    alert.escalated.add(esc["to"])
                    sent.append((esc["to"], "escalated", fp))
        return sent

    def _silenced(self, labels: dict, now: int) -> bool:
        return any(
            start <= now < end and matches(m, labels) for m, start, end in self._silences.values()
        )

Choices worth saying out loud:

• Rejecting an empty silence. An empty matcher set matches every alert. In production that is the "someone muted everything during an incident" outage. Refuse it; a deliberate global mute can be a separate, audited call.
• Receivers deduped per alert, not per route. Two routes with the same receiver (a continue route and a later one) must not page the same person twice for one fire.
• Returning notifications instead of sending them. The router decides; a sender delivers. That split is what makes it testable, and it is where retries and rate limits per receiver go later.

Let R be the number of routes, S the number of silences and A the number of active alerts.

• route: O(R × m), m = matchers per route.
• fire: O(R × m + S) — the silence check scans every silence. With thousands of silences, index them by one label (most silences name alertname or host) and scan only the candidates.
• tick: O(A × (R + S)). At 10,000 active alerts, 100 routes and a tick every 10 seconds that is about 10^6 matcher checks per tick, which is fine in-process. Caching each alert's matching routes on the Alert (routes rarely change) cuts it to O(A).

Worked numbers for "is the dedupe worth it": 2,000 hosts evaluated every 30 s with a disk alert firing on 5% of them sends 100 fire calls per cycle, 12,000 per hour. With a 1-hour repeat_interval the on-call channel gets 100 messages in that hour instead of 12,000.

• "Group alerts." A rack loses power and 40 HostDown alerts fire in the same second. Add group_by: ["alertname", "rack"] to a route. Alerts with the same group key collect into one notification: wait group_wait (say 30 s) after the first one so the rest can join, then send one message listing all of them, and send updates no more often than group_interval (5 min). The dedupe state moves from per alert to per (route, group key).
• "Inhibit." When DatacenterDown fires for dc=us-east, suppress every HostDown with the same dc. An inhibition rule is (source matchers, target matchers, labels that must be equal). Check it at send time, like a silence that is switched on by another alert.
• "Flapping." An alert that resolves and fires every 2 minutes pages every 2 minutes with this code, because resolve clears the dedupe state. Keep resolved state for a short hold time (say 5 min) and treat a re-fire within it as the same incident, or require the condition to hold for N cycles before the first page (for: 2m in the alert rule).
• "Run two replicas for high availability." Two routers must not both page. Options: one active leader with a lease (simple, a failover gap of one lease length); or both active, sharing the notification log (who was told what, when) through gossip or a small store, with each replica waiting a few seconds × its index before sending so the other's log entry arrives first. Alertmanager uses the second; it accepts an occasional duplicate page in exchange for never missing one. Say which way you lean: for paging, a duplicate beats a miss.
• "Delivery fails." The pager API returns 503. Retry with backoff and jitter in the sender, keyed by (receiver, fingerprint, kind) so retries stay idempotent, and alert on the alerting pipeline itself with a dead man's switch: an always-firing alert whose absence pages through a separate path.
• "Regex matchers." Allow {"host": "~db[0-9]+"}. Compile once when the route is loaded, never per alert, and anchor the pattern so db1 does not match proddb10.

Mid-level (L4)

• Correct first-match routing with continue and a default receiver.
• Dedupes repeat fires with a per-alert, per-receiver last-sent time and sends resolved notifications.
• Clean tests for each part with a fake clock.

Senior (L5)

• Picks the fingerprint as identity and explains why label order must not matter.
• Gets the subtle rules right without prompting: resolved goes only to receivers that were paged, the silence window is half-open, escalation runs from the first page and stops on ack.
• Keeps time-driven work in tick and returns notifications instead of sending them, so the design stays testable as parts are added.

Staff (L6+)

• Brings in grouping, inhibition and flapping as the real sources of pager noise, with concrete windows.
• Designs HA without double pages or missed pages, states which failure they prefer and why, and adds a dead man's switch for the alerting path itself.
• Sizes the work per tick and knows where it stops scaling (silence scans, route matching) and how to index it.