Design a traffic forecasting system

Business goal. Routing and ETAs are only as good as the edge weights they run on. Better short-term traffic forecasts mean better routes, more accurate pickup and drop-off ETAs, fewer cancellations from blown ETAs, and better dispatch and pricing, which all consume ETAs.

ML task. Every minute, for every directed road segment in served cities, predict the travel pace (seconds per meter) at horizons h = 5, 10, 15, 30, 45 and 60 minutes, with P10/P50/P90 quantiles. The consumer is the routing engine, not an end user. A route that reaches segment s at time t0 + τ reads the forecast for the horizon nearest τ, interpolated. Beyond 60 minutes it falls back to the historical profile.

Why pace, not speed. Route travel time is a sum of length × pace. Averaging speeds overweights fast traversals (harmonic mean problem), and an error of 5 km/h at 10 km/h costs far more travel time than at 100 km/h. Train on log pace or segment travel time so the loss lines up with ETA error.

Scale and constraints.

• About 1M concurrently online drivers and couriers ping every ~4 s, giving ~250k GPS points/s at peak. That is ~21.6B points/day, or ~2.2 TB/day at ~100 bytes per point.
• ~20M directed segments in served regions. At peak only ~3M have a traversal in the last 15 minutes. Coverage is sparse and very skewed toward arterials.
• Freshness: an observation should affect published weights within ~2 minutes (ingest to snapshot).
• Routing p99 is ~100 ms per query at hundreds of thousands of QPS. The model is not called per request. Forecasts are precomputed into weight snapshots that the routing servers load.

ML objective vs business metric. Offline objective: traversal-weighted error on log pace per horizon, plus quantile calibration. Business metrics: trip-level ETA MAE and the share of trips with |ETA error| > 20%, route quality (realized trip duration on the route chosen), and downstream effects on cancellations and conversion. The weighting matters. Unweighted segment MAE is dominated by millions of empty residential streets that no route uses.

Sources.

• Raw GPS from the driver app: lat/lng, accuracy, heading, speed, timestamp, trip state (en route, on trip, idle).
• Road graph: segment geometry, length, road class, speed limit, lane count, turn restrictions, and stable segment IDs with version history.
• Completed trips with actual pickup and drop-off times, used for route-level evaluation.
• Context: incident and closure feeds, weather, the event calendar, and the holiday calendar.

Map matching. An HMM map matcher (Viterbi over candidate segments, with emission cost from GPS distance and transition cost from the difference between route distance and great-circle distance) turns pings into segment traversals with entry and exit timestamps. This runs on the stream, keyed by driver, with a ~30 s window. Traversals that cut across a segment are split by length fraction.

Cleaning and bias.

• Drop idle and parked drivers, and drop dwell during pickup or drop-off. Otherwise a 90 s curbside wait looks like a jam.
• Drop GPS with accuracy > 30 m and drop urban-canyon jumps. Cap pace at physical limits.
• Aggregate per (segment, 1-minute bucket): count, harmonic-mean speed (mean pace), and pace variance. Weight each driver once so one slow driver looping a block doesn't dominate.
• Fleet bias: ride-hail drivers stop more and run at different hours than general traffic. The model learns its own calibration because labels come from the same fleet. Note this as a caveat when ETAs are consumed by non-fleet users.

Label. y(s, t, h) = the mean pace of traversals of segment s that enter during [t + h − 2.5 min, t + h + 2.5 min]. If there are none, the label is missing and the row is masked (no imputation). Label delay equals h plus segment traversal time plus ~1 minute of pipeline lag. The 60-minute label is ready about 62 minutes after the forecast, so near-real-time accuracy monitoring is possible.

Imbalance. Most segment-minutes are free flow. The forecasts that matter are the transitions: congestion onset and clearance, incidents, and event egress. Upweight rows where the realized pace differs from the historical profile by more than 30%, and report those slices separately.

Leakage traps.

• Features at forecast time t may only use traversals that exited before t minus the pipeline lag, not those that entered before t. A traversal still in progress at t is not known yet.
• Historical profiles must be computed only from data before the training window. A profile built over the whole dataset leaks the test period's congestion.
• Smoothing or imputation filters must be causal. No centered rolling means.
• Map changes: a new segment ID created after t must not appear in features.
• Random row splits leak strongly through temporal and spatial autocorrelation.

Splits and sampling. Use rolling-origin backtests: train on 8 weeks, validate on the following week, test on the week after that, then roll forward. Include holiday weeks in test explicitly. Subsample free-flow minor-road rows about 10:1 and keep their weights. For cold-start evaluation, hold out whole new cities.

Real-time (streaming, refreshed every minute).

• Segment pace in the last 1, 5 and 15 minutes, observation counts, and minutes since the last observation.
• Pace trend: slope over the last 15 minutes.
• Upstream and downstream neighbors (1 and 2 hops on the directed graph): pace and counts over the last 5 minutes. Congestion propagates upstream at ~15–20 km/h, so the downstream state is a leading indicator.
• Tile-level aggregates (~1 km H3-style cell): mean pace ratio to profile, and active driver density.
• Live incident and closure flags within 2 hops, with minutes since reported.

Historical (batch, nightly).

• Profile pace by segment × day-of-week × 15-minute slot, as median and P85. Recency-weighted with a 4-week half-life and shrunk toward the road-class prior when counts are low.
• Same slot last week, and the deviation of today's last hour from the profile. Persistent deviations predict persistent deviations.

Static. Road class, speed limit, length, lanes, signals per km, turn type at the end node, a learned segment embedding (only for segments with enough data), and a city ID.

Context. Forecast horizon h as an input when one model serves all horizons, target time-of-day and day-of-week, holiday flag, weather (precipitation intensity, visibility), and the scheduled event end time near the segment.

Missing data. About 85% of segments have no fresh observation at a given minute. Encode staleness explicitly (minutes since last observation, observation count) rather than forward-filling silently. The model learns to fall back to the profile when the evidence is stale.

Feature store. Streaming aggregates go to a low-latency KV keyed by (segment, minute), and batch profiles go to an offline store. Both are produced from the same definitions so training features are point-in-time joined from logs. Training uses the logged feature snapshot as it was at t, not a recomputation, which avoids training/serving skew from late events.

Baselines (must beat both, per horizon).

1. Persistence: pace now = pace at t + h. Strong at 5 minutes, weak at 60.
2. Historical profile at slot(t + h). Strong at 60 minutes on normal days, useless during incidents.
3. Blend: w(h, freshness) × persistence + (1 − w) × profile, with w fit per road class. This is often within 10–15% of a much fancier model and is the fallback when the model service is down.

Production v1: GBDT residual model. Predict r = log pace(t + h) − log profile(slot(t + h)), with h as a feature, using one model per region. Predicting the residual means the model only learns deviations, and it degrades gracefully to the profile when the evidence is stale. Training uses a Huber loss on the residual, weighted by expected route usage (traversal volume over 4 weeks). Separate quantile heads (pinball loss at τ = 0.1, 0.5, 0.9) give the uncertainty band that routing uses for risk-averse ETAs. Inference cost: 3M active segments × 6 horizons = 18M rows per minute, or ~300k rows/s. That is a few dozen cores for a 300-tree model. Stale segments (no observation in 30 minutes) skip the model and use the profile plus the tile-level deviation.

Production v2: spatio-temporal graph model. The GBDT sees neighbors only through hand-built 1–2 hop features. A diffusion-convolution or graph-attention encoder over the road graph (2–3 hops), feeding a temporal encoder (GRU or temporal conv over the last 30 minutes) and a multi-horizon decoder, captures how congestion propagates and how jams clear across junctions. Run it per tile partition (~50k segments plus a halo of 3 hops so boundary segments see their neighbors). The loss is masked Huber on log-pace residuals, summed across horizons with weights, and computed only where labels exist. Ship it only where it beats the GBDT on the transition slices. It is costlier to train and harder to debug.

Post-processing.

• Clamp to [free-flow pace × 0.9, closure threshold]. Mark closed segments as impassable instead of predicting a huge pace.
• Make horizons monotone in time near known event ends where that applies. Otherwise leave them unconstrained.
• Shrink toward the profile when the observation count is low (an empirical-Bayes style blend).

Offline, per horizon and per city.

• Traversal-weighted MAE and MAPE on segment travel time, and the skill score vs persistence and vs profile: 1 − MAE_model / MAE_baseline.
• Slices: free-flow vs transitions (|realized − profile| > 30%), incident-affected segments, road class, data freshness bucket, holidays, rain.
• Quantile calibration: the P10–P90 interval should cover about 80% of traversals, per horizon, with a pinball loss report.
• Route-level replay (the metric that matters). For held-out completed trips, rebuild the route the driver took. Sum the forecast travel times using the forecast made at trip start, read at the time each segment was reached. Compare with the actual trip duration: ETA MAE, MAPE, and the share of trips with error > 20%. Also re-run routing on the forecast weights and check whether the chosen routes get faster against realized weights. Segment metrics can improve while route metrics don't, because errors on unused segments don't matter and correlated errors along a route add up.

Online.

• Randomize by city and time block (switchback, e.g. 2-hour blocks), not by rider. Routes change where traffic flows, and dispatch shares drivers, so rider-level randomization contaminates both arms.
• Primary metrics: trip-level ETA MAE for pickup and trip ETAs, the share of trips with |error| > 20%, and realized trip duration.
• Secondary metrics: rider cancellation after request, driver reroute rate, and conversion on quoted ETAs.
• Guardrails: routing p99 latency, snapshot freshness, the rate of routes through closed roads, driver complaints on bad routes, and ETA bias (systematic underestimates hurt trust more than overestimates).
• Run 2–4 weeks so that weekday and weekend patterns are both covered.

The online path, from pings to published weights:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"app","x":0,"y":140,"width":200,"height":70,"label":{"text":"Driver app GPS\npings, 250k/s"}},
 {"type":"rectangle","id":"kafka","x":300,"y":140,"width":180,"height":70,"label":{"text":"Kafka"}},
 {"type":"rectangle","id":"mm","x":560,"y":140,"width":200,"height":70,"label":{"text":"Map matcher,\nHMM per driver"}},
 {"type":"rectangle","id":"agg","x":860,"y":140,"width":200,"height":70,"label":{"text":"Segment-minute\naggregator"}},
 {"type":"ellipse","id":"fs","x":1160,"y":130,"width":220,"height":90,"label":{"text":"Online feature\nstore"}},
 {"type":"rectangle","id":"fw","x":1480,"y":140,"width":220,"height":70,"label":{"text":"Forecast workers\nper tile, every 60s"}},
 {"type":"rectangle","id":"prof","x":920,"y":-90,"width":180,"height":70,"label":{"text":"Historical\nprofiles, nightly"}},
 {"type":"rectangle","id":"inc","x":1180,"y":-90,"width":200,"height":70,"label":{"text":"Incidents,\nweather, events"}},
 {"type":"rectangle","id":"reg","x":1500,"y":-90,"width":180,"height":70,"label":{"text":"Model registry"}},
 {"type":"ellipse","id":"snap","x":1470,"y":330,"width":240,"height":90,"label":{"text":"Weight snapshot\nstore, versioned"}},
 {"type":"rectangle","id":"route","x":1120,"y":340,"width":220,"height":70,"label":{"text":"Routing servers,\nCCH re-customize"}},
 {"type":"rectangle","id":"eta","x":800,"y":340,"width":200,"height":70,"label":{"text":"ETA service\nand dispatch"}},
 {"type":"arrow","id":"app-kafka","x":200,"y":175,"width":100,"height":0,"start":{"id":"app"},"end":{"id":"kafka"}},
 {"type":"arrow","id":"kafka-mm","x":480,"y":175,"width":80,"height":0,"start":{"id":"kafka"},"end":{"id":"mm"}},
 {"type":"arrow","id":"mm-agg","x":760,"y":175,"width":100,"height":0,"start":{"id":"mm"},"end":{"id":"agg"}},
 {"type":"arrow","id":"agg-fs","x":1060,"y":175,"width":100,"height":0,"start":{"id":"agg"},"end":{"id":"fs"}},
 {"type":"arrow","id":"prof-fs","x":1050,"y":-20,"width":174,"height":154,"start":{"id":"prof"},"end":{"id":"fs"}},
 {"type":"arrow","id":"inc-fs","x":1278,"y":-20,"width":-6,"height":150,"start":{"id":"inc"},"end":{"id":"fs"}},
 {"type":"arrow","id":"fs-fw","x":1380,"y":175,"width":100,"height":0,"start":{"id":"fs"},"end":{"id":"fw"}},
 {"type":"arrow","id":"reg-fw","x":1590,"y":-20,"width":1,"height":160,"start":{"id":"reg"},"end":{"id":"fw"}},
 {"type":"arrow","id":"fw-snap","x":1590,"y":210,"width":1,"height":120,"start":{"id":"fw"},"end":{"id":"snap"}},
 {"type":"arrow","id":"snap-route","x":1470,"y":375,"width":-130,"height":0,"start":{"id":"snap"},"end":{"id":"route"}},
 {"type":"arrow","id":"route-eta","x":1120,"y":375,"width":-120,"height":0,"start":{"id":"route"},"end":{"id":"eta"}}
]}

The training and accuracy loop:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"agg","x":0,"y":140,"width":200,"height":70,"label":{"text":"Segment-minute\naggregator"}},
 {"type":"ellipse","id":"fs","x":0,"y":320,"width":220,"height":90,"label":{"text":"Online feature\nstore"}},
 {"type":"rectangle","id":"labels","x":320,"y":140,"width":180,"height":70,"label":{"text":"Traversal\nlabel log"}},
 {"type":"rectangle","id":"snaps","x":320,"y":330,"width":180,"height":70,"label":{"text":"Feature\nsnapshot log"}},
 {"type":"rectangle","id":"join","x":620,"y":235,"width":180,"height":70,"label":{"text":"Point-in-time\njoin"}},
 {"type":"rectangle","id":"train","x":900,"y":225,"width":240,"height":90,"label":{"text":"Training: GBDT daily,\ngraph model weekly"}},
 {"type":"rectangle","id":"gate","x":1240,"y":225,"width":220,"height":90,"label":{"text":"Backtest and route\nreplay gate"}},
 {"type":"rectangle","id":"reg","x":1540,"y":235,"width":180,"height":70,"label":{"text":"Model registry"}},
 {"type":"rectangle","id":"fw","x":940,"y":-20,"width":220,"height":70,"label":{"text":"Forecast workers\nper tile, every 60s"}},
 {"type":"rectangle","id":"acc","x":600,"y":-20,"width":220,"height":70,"label":{"text":"Live accuracy\nmonitor per horizon"}},
 {"type":"arrow","id":"agg-labels","x":200,"y":175,"width":120,"height":0,"start":{"id":"agg"},"end":{"id":"labels"}},
 {"type":"arrow","id":"fs-snaps","x":220,"y":365,"width":100,"height":0,"start":{"id":"fs"},"end":{"id":"snaps"}},
 {"type":"arrow","id":"labels-join","x":500,"y":204,"width":120,"height":38,"start":{"id":"labels"},"end":{"id":"join"}},
 {"type":"arrow","id":"snaps-join","x":500,"y":336,"width":120,"height":-38,"start":{"id":"snaps"},"end":{"id":"join"}},
 {"type":"arrow","id":"join-train","x":800,"y":270,"width":100,"height":0,"start":{"id":"join"},"end":{"id":"train"}},
 {"type":"arrow","id":"train-gate","x":1140,"y":270,"width":100,"height":0,"start":{"id":"train"},"end":{"id":"gate"}},
 {"type":"arrow","id":"gate-reg","x":1460,"y":270,"width":80,"height":0,"start":{"id":"gate"},"end":{"id":"reg"}},
 {"type":"arrow","id":"reg-fw","x":1550,"y":235,"width":-420,"height":-185,"start":{"id":"reg"},"end":{"id":"fw"}},
 {"type":"arrow","id":"labels-acc","x":476,"y":140,"width":168,"height":-90,"start":{"id":"labels"},"end":{"id":"acc"}},
 {"type":"arrow","id":"fw-acc","x":940,"y":15,"width":-120,"height":0,"start":{"id":"fw"},"end":{"id":"acc"}}
]}

Online path and latency budget (ingest to published weights is about 120 s or less).

• Kafka plus map matching: ~30 s, which is the matcher window.
• Aggregation into minute buckets: ~15 s after the minute closes.
• Feature fetch and inference per tile: ~20 s for all tiles in parallel.
• Snapshot write: ~10 s. 20M segments × 6 horizons × 2 bytes (quantized pace) is ~240 MB per snapshot, and deltas usually run under 30 MB.
• Routing servers pull deltas and re-customize customizable contraction hierarchies (CCH) in ~5–10 s per region.
• The query path only reads weights, so routing p99 stays under 100 ms no matter how big the model is.

Fallbacks. If a snapshot is stale for more than 5 minutes, routing switches to the blend baseline computed locally from the profile. A circuit breaker rejects snapshots whose median pace shifts by more than 25% city-wide from one minute to the next.

Monitoring.

• Pipeline: ingest lag, ping rate per city (an app release can halve it), map-match failure rate, and the share of segments with fresh observations.
• Accuracy: live MAE per horizon and city, computed as labels mature (5 to 62 minutes later), with alerts on skill score falling below persistence.
• Drift: feature distributions (pace ratios, GPS accuracy) and graph changes (new or split segment IDs remapped by geometry overlap).

Feedback loops.

• Routing sends drivers onto roads predicted to be fast. That changes those roads' traffic and means the fleet mostly observes roads it chose (selection bias).
• Mitigations: the profile prior on unobserved roads, a small share of trips routed on near-tied alternatives to keep observing, and the fact that labels come from all fleet traversals, not just routed ones.

Retraining. Profiles nightly. GBDT daily per region on a rolling 8 weeks. Graph model weekly. Every model must beat the incumbent on route replay before rollout, then goes through shadow scoring and canary cities. Holidays and big events get explicit calendar features, because one year of history has only one Thanksgiving.

• Precomputed snapshots vs per-request inference. Snapshots keep routing fast and simple, but they freeze forecasts for up to ~2 minutes and can't condition on the specific route or vehicle.
• Pace target vs speed target. Pace aligns with travel-time error, but it is unbounded at standstill and needs clamping plus a separate closure state.
• Residual over profile vs raw prediction. The residual gives graceful degradation and an easier model, but it inherits profile errors on recently changed roads (new signals, construction).
• GBDT vs graph model. The GBDT is cheap, debuggable and fast to retrain, but it sees propagation only through hand-built neighbor features. The graph model captures propagation but costs more to train and serve, is harder to explain, and is harder to partition at tile boundaries.
• One multi-horizon model vs one model per horizon. A single model shares data and is simpler to operate, but it may underfit at long horizons where the features matter differently.
• Fleet-only GPS. Free and fresh, but sparse and biased off arterials. Adding external probe data improves coverage at the cost of money, latency and harder label consistency.
• Switchback A/B. It handles interference but has lower statistical power and needs longer tests than user-level randomization.
• Quantile outputs. Quantiles allow risk-aware ETAs and buffers but triple the stored weights, and routing has to decide which quantile to optimize.

Mid-level (L4)

• Frames the problem as per-segment, multi-horizon regression, and gets from raw GPS to labels via map matching and aggregation.
• Proposes persistence and historical-profile baselines, then a GBDT with recent-pace, time-of-day and neighbor features.
• Uses a time-based split and an MAE metric, and knows that forecasts are precomputed rather than computed per routing request.

Senior (L5)

• Predicts pace or travel time, not speed, and weights the loss by route usage. Adds route-level replay as the key offline metric.
• Names the leakage traps: in-progress traversals, non-causal profiles, autocorrelated random splits. Handles sparsity with staleness features and shrinkage to the profile.
• Sizes ingest, inference and snapshots, budgets the ~2-minute freshness path, and plans fallbacks and live accuracy monitoring as labels mature.
• Uses switchback experiments because routing changes traffic.

Staff (L6+)

• Positions the system inside the stack: time-dependent routing weights, the downstream ETA correction model, dispatch and pricing. Defines who owns which error.
• Argues when a spatio-temporal graph model is worth its cost, using transition-slice evidence, and how to partition it at tile boundaries.
• Addresses the feedback loops and selection bias from routing, map versioning, and holiday and event cold start. Uses quantiles for risk-aware ETAs.
• Sets rollout gates (route replay, shadow scoring, canary cities, circuit breakers) and cost targets per city.