0

Design a customer churn prediction system

by Frontier Interview· 1 day ago· ⑂ 0 remixes

Question: Design a customer churn prediction system

Companies: Stripe

Difficulty: Medium

Problem statement

Predict which business customers will stop using the product in the next 30 days, so account teams can act in time.

Problem framing

  • Business goal: keep revenue from business customers (merchants) by letting account managers and success teams reach at-risk accounts while there is still time to save them. Stripe earns on processed volume, so the loss that matters is revenue lost, not the number of logos lost.
  • ML task: each day, for every active business account, predict P(the account churns in the next 30 days). Rank accounts by expected revenue at risk = P(churn) x trailing annualized net revenue. Each account in the list carries reason codes the account manager can act on.
  • Who consumes it: about 50k managed accounts have an account manager. The long tail (~2M self-serve accounts) gets automated nudges such as emails, in-dashboard prompts and integration-health warnings. The team has fixed capacity: about 2,500 meaningful outreaches per month.
  • Constraints: daily batch is enough for the main list. Scores must be in the CRM by 06:00 in each region. A small streaming path flags acute events (volume falls off a cliff, an integration breaks) within minutes. Scoring 2M rows with a GBDT takes a few minutes, so latency is not the hard part. Label delay, feedback loops and actionability are.
  • ML objective vs business metric: offline, the target is calibrated log loss and revenue-weighted precision/recall at the team's capacity K. The business metric is net revenue retained at 90 days in treated accounts vs a randomized holdout. A model that predicts churn well but flags accounts nobody can save produces no value.

Data and labels

  • Sources: the payments ledger (charges, refunds, disputes, payouts, by day and product), API request logs (endpoints, error rates, API version, webhook failures), dashboard and product usage events, billing for Stripe's own fees (failed invoices, plan changes), support tickets and chat transcripts, CRM data (contract end date, account owner, last touch, negotiated pricing), firmographics (industry/MCC, country, company age, size band) and account lifecycle events (closures, deactivated API keys).
  • Churn definition. Usage-based churn rarely shows up as a cancellation. Merchants move volume to another processor and leave the account open. Define it per snapshot date t for accounts that are active at t (>= 20 charges in each of the last 8 weeks):
    • Hard churn: the account closes, or volume over [t, t+30) is < 10% of the trailing 90-day daily average x 30.
    • Partial churn (secondary head): volume over [t, t+30) is < 50% of baseline after a seasonal adjustment.
    • Seasonality: compare against the same window last year scaled by YoY growth. Otherwise a ski resort looks like it churns every April. Exclude accounts tagged seasonal when they have less than one year of history.
  • Label delay: the 30-day window ends at t+30, and closures and refunds settle within a few more days. A snapshot gets a final label at about t+35. The newest trainable snapshot is therefore always about 5 weeks old.
  • Imbalance: about 2% of managed accounts hard-churn in a month, about 1,000 of 50k. Partial churn is about 5%. Use all positives, downsample negatives to about 1:10 per snapshot, and correct the predicted probabilities for the sampling rate so calibration holds.
  • Snapshots: each row is (account, snapshot date). Take weekly snapshots over 24 months, so the same account appears many times. Cap rows per account and use the account ID as the grouping key so that no account is split across folds within one time window.
  • Leakage traps:
    • Any feature computed with data at or after t. All features come from a point-in-time feature store with as_of = t.
    • Signals that are the churn itself: "account closure started", "API keys revoked", "payout bank account removed". Leave them out of the model and send them through the streaming alert path, because they arrive too late to act on.
    • CRM fields filled in after the fact, such as "churn reason" or a "lost" opportunity stage.
    • Outreach history: accounts that were contacted are biased by the previous model's selection (see feedback loops).
  • Splits: split strictly by time. Train on snapshots up to month M-2. Leave a 30-day gap so that training labels do not overlap the validation window. Validate on month M-1 and test on month M. Backtest on 6 rolling origins to check stability across seasons such as Q4 retail peaks.

Features

  • Volume trend (most predictive): gross volume and charge count over 7/28/90 days. Ratios 7d/90d and 28d/prior-28d. Slope of a linear fit over 12 weeks. YoY ratio. Share of days with zero volume. Average ticket size.
  • Processor-switch proxies: a falling mix of payment methods. Authorization attempts drop while the merchant's own traffic signals stay flat. A sudden fall in a single product line, for example Billing renewals stop while one-off charges continue.
  • Payment health: authorization rate and its change, decline-code mix, dispute and fraud rates, Radar block rate. A spike in false declines frustrates merchants.
  • Integration health: API error rate (4xx/5xx), webhook delivery failures, months behind the current API version, deprecated endpoints in use, failed deploys (bursts of new errors).
  • Product depth: number of products adopted (Payments, Billing, Connect, Terminal, Tax). Connected accounts count. Seats on the team. Deep integrations raise switching cost and lower churn.
  • Commercial: effective take rate vs segment median, a recent price increase, days until contract end, failed payment of Stripe invoices, volume commitment shortfall.
  • Engagement and support: dashboard logins over 28 days and their trend, admin users removed, ticket count, unresolved tickets older than 7 days, a sentiment score and topic tags from an LLM classifier on tickets ("pricing", "competitor", "payout delay", "migration"), and data-export events.
  • Firmographic: industry, country, size band, account age, Connect platform vs direct merchant.
  • Online vs offline: almost everything is a daily batch aggregate from the warehouse (Spark to an offline store, materialized to an online KV store for the CRM and the dashboard). The streaming path (Flink on the event bus) keeps only minute-level counters: hourly volume vs expected, API error bursts, closure-flow page views. These counters drive rule-based alerts, not the model.
  • Freshness SLA: daily features as of 00:00 UTC, ready by 03:00. A late partition blocks scoring for that segment, and yesterday's scores stay in place with a staleness flag.

Model

  • Baseline 1 (rule): flag an account when its 28d/prior-28d volume falls below 0.7 or its tickets with "competitor" tags go above 0. Simple, explainable, and the bar to beat.
  • Baseline 2: L2-regularized logistic regression on about 40 standardized features. It is calibrated out of the box and its coefficients double as a sanity check.
  • Production: a LightGBM binary classifier on about 300 features. It handles missing values, monotone constraints (for example, risk must not fall as the volume ratio falls) and heterogeneous scales. Loss is binary cross-entropy with sample weights of log(1 + annual_revenue). With raw revenue weights, a few giant accounts would dominate the gradient. With no weights, the model would optimize for small accounts that matter little.
  • Multi-horizon option: a discrete-time hazard model (one row per account-week, target = churned this week) gives 30/60/90-day curves from one model. That helps for accounts with renewal dates 60 days out.
  • Calibration: isotonic regression on the validation month, because expected revenue at risk multiplies the probability by revenue. Correct for negative downsampling first: p = p_s / (p_s + (1 - p_s) / r), where r is the negative sampling rate.
  • Reason codes: compute TreeSHAP per account. Map the top 3 positive contributors to human templates, for example "Volume down 42% vs prior 28 days" or "Webhook failures up 8x since Sep 30". Suppress reasons the account manager cannot act on, such as industry.
  • Next step, uplift: risk alone does not decide whom to contact. Once randomized holdout data exists (below), train a T-learner or an uplift GBDT for P(retain | contacted) - P(retain | not contacted), and rank by uplift x revenue. Accounts that are certain to churn, and accounts that never needed help, both drop down the list.

Evaluation

  • Offline, primary: revenue-weighted recall at K = 2,500 accounts per month, i.e. the share of churned revenue that appears in the top-K list ranked by expected revenue at risk. Also precision@K. At a 2% base rate, a random list of 2,500 holds about 50 churners. The volume-drop rule reaches about 12% precision (300). The target for the GBDT is >= 20% precision (500 churners, 50% count recall, 10x lift).
  • Offline, secondary: PR-AUC (ROC-AUC looks flattering at a 2% base rate), log loss, and calibration measured as ECE and reliability curves per segment. The predicted churn count per decile should be within 10% of the actual count.
  • Lead time: median days between the first flag and the churn event. A flag 2 days before the volume reaches zero catches the churn but leaves no time to act. Require >= 14 days median lead time.
  • Slices: size band, industry, country, Connect platforms vs direct merchants, account age < 6 months. A model that only catches small accounts can still score well on count metrics.
  • Online A/B: randomize at the account level. Hold out 10% of the flagged top-K from proactive outreach. The primary metric is net revenue retained at 90 days, treated vs holdout. Secondary metrics are 30-day hard churn rate and volume recovery. At about 2,500 flagged per month, holding out 250 per month takes about 3 months to detect a 5-point absolute churn drop at 20% baseline. Pool months and use CUPED on the pre-period volume.
  • Guardrails: account manager load (accounts per person per week), the share of flags the account manager marks "not useful" in the CRM, CSAT and NPS for contacted accounts, and churn of contacted accounts that were not at risk (outreach can remind a merchant to leave).

Serving and monitoring

Daily batch scoring, plus the streaming path for acute alerts:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"src","x":0,"y":0,"width":200,"height":90,"label":{"text":"Ledger, API logs,\ntickets, CRM"}},
 {"type":"rectangle","id":"wh","x":280,"y":0,"width":200,"height":90,"label":{"text":"Warehouse daily\npartitions"}},
 {"type":"rectangle","id":"spark","x":560,"y":0,"width":210,"height":90,"label":{"text":"Spark feature jobs,\npoint-in-time"}},
 {"type":"ellipse","id":"fs","x":850,"y":0,"width":220,"height":90,"label":{"text":"Offline\nfeature store"}},
 {"type":"rectangle","id":"scorer","x":1150,"y":0,"width":200,"height":90,"label":{"text":"Batch scorer:\nLightGBM +\ncalibrator"}},
 {"type":"rectangle","id":"scores","x":1430,"y":0,"width":200,"height":90,"label":{"text":"Scores, revenue\nat risk, SHAP\nreasons"}},
 {"type":"rectangle","id":"crm","x":1420,"y":240,"width":220,"height":90,"label":{"text":"CRM sync and\naccount manager\nqueue"}},
 {"type":"rectangle","id":"nudge","x":1710,"y":0,"width":220,"height":90,"label":{"text":"Self-serve nudges:\nemail, dashboard"}},
 {"type":"rectangle","id":"flink","x":280,"y":240,"width":200,"height":90,"label":{"text":"Flink streaming\ncounters"}},
 {"type":"rectangle","id":"rules","x":560,"y":240,"width":200,"height":90,"label":{"text":"Acute alert\nrules"}},
 {"type":"arrow","id":"src-wh","x":200,"y":45,"width":80,"height":0,"start":{"id":"src"},"end":{"id":"wh"}},
 {"type":"arrow","id":"wh-spark","x":480,"y":45,"width":80,"height":0,"start":{"id":"wh"},"end":{"id":"spark"}},
 {"type":"arrow","id":"spark-fs","x":770,"y":45,"width":80,"height":0,"start":{"id":"spark"},"end":{"id":"fs"}},
 {"type":"arrow","id":"fs-scorer","x":1070,"y":45,"width":80,"height":0,"start":{"id":"fs"},"end":{"id":"scorer"}},
 {"type":"arrow","id":"scorer-scores","x":1350,"y":45,"width":80,"height":0,"start":{"id":"scorer"},"end":{"id":"scores"}},
 {"type":"arrow","id":"scores-crm","x":1530,"y":90,"width":1,"height":150,"start":{"id":"scores"},"end":{"id":"crm"}},
 {"type":"arrow","id":"scores-nudge","x":1630,"y":45,"width":80,"height":0,"start":{"id":"scores"},"end":{"id":"nudge"}},
 {"type":"arrow","id":"src-flink","x":100,"y":90,"width":180,"height":195,"start":{"id":"src"},"end":{"id":"flink"}},
 {"type":"arrow","id":"flink-rules","x":480,"y":285,"width":80,"height":0,"start":{"id":"flink"},"end":{"id":"rules"}},
 {"type":"arrow","id":"rules-crm","x":760,"y":285,"width":660,"height":0,"start":{"id":"rules"},"end":{"id":"crm"}}
]}

Outcomes become labels for the monthly retrain:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"crm","x":0,"y":0,"width":220,"height":90,"label":{"text":"CRM sync and\naccount manager\nqueue"}},
 {"type":"rectangle","id":"out","x":300,"y":0,"width":220,"height":90,"label":{"text":"Outcomes: contacted,\nholdout, churned\nat t+35"}},
 {"type":"rectangle","id":"labels","x":600,"y":0,"width":180,"height":90,"label":{"text":"Label builder"}},
 {"type":"ellipse","id":"fs","x":850,"y":-200,"width":220,"height":90,"label":{"text":"Offline\nfeature store"}},
 {"type":"rectangle","id":"retrain","x":860,"y":0,"width":200,"height":90,"label":{"text":"Monthly retrain\nand backtest"}},
 {"type":"rectangle","id":"reg","x":1140,"y":0,"width":200,"height":90,"label":{"text":"Model registry,\nshadow, promote"}},
 {"type":"rectangle","id":"scorer","x":1420,"y":0,"width":200,"height":90,"label":{"text":"Batch scorer:\nLightGBM +\ncalibrator"}},
 {"type":"arrow","id":"crm-out","x":220,"y":45,"width":80,"height":0,"start":{"id":"crm"},"end":{"id":"out"}},
 {"type":"arrow","id":"out-labels","x":520,"y":45,"width":80,"height":0,"start":{"id":"out"},"end":{"id":"labels"}},
 {"type":"arrow","id":"labels-retrain","x":780,"y":45,"width":80,"height":0,"start":{"id":"labels"},"end":{"id":"retrain"}},
 {"type":"arrow","id":"fs-retrain","x":960,"y":-110,"width":1,"height":110,"start":{"id":"fs"},"end":{"id":"retrain"}},
 {"type":"arrow","id":"retrain-reg","x":1060,"y":45,"width":80,"height":0,"start":{"id":"retrain"},"end":{"id":"reg"}},
 {"type":"arrow","id":"reg-scorer","x":1340,"y":45,"width":80,"height":0,"start":{"id":"reg"},"end":{"id":"scorer"}}
]}
  • Latency budget (batch): partitions land by 01:00 UTC. Features run 01:00-03:00. Scoring of 2M rows plus SHAP takes about 15 minutes on a small cluster. CRM sync runs 03:30-05:00 under Salesforce API rate limits, so send only accounts whose score or rank changed by more than a threshold. Streaming alerts reach the account manager queue in < 5 minutes.
  • Drift: daily PSI on the top 30 features and on the score distribution. Alert at PSI > 0.2. Watch for upstream changes such as a new product line or a renamed decline code that silently break features. Feature null rate is checked against a 7-day baseline.
  • Delayed performance tracking: each snapshot's realized precision@K and calibration are computed at t+35 and plotted as a lagging dashboard. Leading proxies are the share of flagged accounts whose volume keeps falling over the next 7 days, and account manager feedback.
  • Feedback loop: successful outreach turns positives into negatives, so retraining on treated accounts teaches the model that these signals are harmless. Mitigations: train risk models on the holdout plus untreated accounts, or add a treated indicator and set it to 0 at inference time. Log every outreach with a timestamp. The randomized holdout must stay permanent.
  • Retraining: monthly, on a rolling 24-month window, with 6-origin backtests. Promote only if revenue-weighted recall@K improves and calibration ECE does not regress on any major segment. Shadow-score for 2 weeks before promotion. Retrain out of cycle after a pricing change or a macro shock.

Trade-offs

  • Daily batch vs real-time scoring: batch is cheap, reproducible and enough for a 30-day horizon. It gives up same-day reaction, which the rule-based streaming alerts cover for acute cases.
  • Volume-based churn label vs account closure: it catches silent migrations, which make up most real churn. It gives up a crisp ground truth, and the thresholds (10%, 50%) and seasonal adjustment need careful tuning.
  • Ranking by revenue at risk vs by probability: it aligns with dollars and the team's capacity. It gives up coverage of small accounts, which go to automated nudges instead.
  • log(revenue) weights vs raw revenue weights: stable training. It underweights the very largest accounts, so give the top 1% of accounts a dedicated review regardless of score.
  • GBDT vs sequence models (transformer over daily volume): GBDT is fast, explainable with SHAP and strong on tabular data. Hand-built trend features may miss shape patterns a sequence model would learn. Try a sequence model only once the GBDT plateaus.
  • Risk model vs uplift model: risk is easy to validate today. Uplift targets the people who can be saved but needs randomized treatment data and has noisy, hard-to-evaluate estimates.
  • Permanent holdout: it gives honest impact measurement and unbiased training labels. The cost is revenue lost in the held-out accounts, so keep it small (10%) and exclude strategic accounts.

Interviewer expectations

Mid-level (L4)

  • Defines churn concretely, including a volume-based version for a usage-based business.
  • Builds snapshot rows with point-in-time features and a time-based split.
  • Proposes logistic regression then GBDT, and handles imbalance with PR-AUC instead of accuracy.

Senior (L5)

  • Ties the metric to account manager capacity: precision and revenue-weighted recall at K, lead time, calibration for expected revenue at risk.
  • Calls out leakage (late signals, CRM fields filled in afterwards), the 30-day label delay and seasonality.
  • Designs reason codes, the CRM integration, drift monitoring and a randomized holdout to measure retained revenue.

Staff (L6+)

  • Recognizes the intervention feedback loop and keeps a permanent holdout to fix both measurement and training labels.
  • Moves the question from "who will churn" to "who can we save": uplift modeling, routing self-serve vs managed accounts.
  • Discusses org fit: account manager trust in reasons, out-of-cycle retrains after pricing changes, and guardrails against outreach that causes churn.

Sign in to comment, vote or remix.