Design a payment fraud detection system

Business goal. Stop card-payment fraud for merchants on a payments platform without declining good customers. Every fraudulent approval costs the merchant the goods, the refunded amount and a dispute fee (~$15). Every false decline costs the sale, the merchant's margin, and often the customer for good. A 1% rise in false declines usually costs more revenue than the fraud it stops.

ML task. For each authorization attempt, predict P(fraud | transaction, context at authorization time). "Fraud" means the cardholder or issuer later disputes the charge as unauthorized. The score is produced synchronously, before the request goes to the card network. A decision layer maps score + amount + merchant policy to one of: allow, step-up (3-D Secure challenge), review (manual queue, for merchants that ship physical goods later), or block.

Constraints.

• Volume: ~2,000 authorizations/s on average, 10,000/s at peak (Black Friday, flash sales). About 170M scored transactions/day.
• Latency: the scoring call sits inside the authorization path. Budget is p99 ≤ 40 ms end to end for risk scoring, out of a ~1 s total auth budget.
• Availability: if scoring fails, fall back to rules plus a cached per-merchant default. Never fail closed for all traffic.
• Base rate: ~0.1% of transactions by count, ~0.3% by value (fraud skews to higher amounts and resaleable goods).
• Labels arrive late: most fraud chargebacks land in 30–60 days; networks allow up to 120 days.

ML objective vs business metric. The model minimizes weighted log loss, so it outputs a calibrated probability. The business metric is expected cost:

• cost(block a good transaction) = merchant margin × amount + customer lifetime loss
• cost(allow fraud) = amount + dispute fee + any network penalty

Block when p × (amount + fee) > (1 − p) × margin × amount. Worked example: $100 order, 30% margin, $15 fee. Block if 115p > 30(1 − p), i.e. p > 30/145 ≈ 0.21. The threshold depends on amount and margin, so calibration matters more than ranking alone. Reported KPIs: fraud loss in basis points of volume, false-positive (good-declined) rate, authorization rate, and dispute rate (card networks put merchants above ~0.9% disputes on monitoring programs).

Sources.

• Authorization events: amount, currency, merchant, MCC, card BIN, entry mode (card-not-present, wallet, card-present), 3DS result, AVS/CVC check results, issuer response.
• Client signals: device fingerprint and browser/JS telemetry from the checkout SDK, IP, user agent, session behavior (time on page, paste events on card field).
• Customer and merchant data: email, shipping/billing address, merchant account age, merchant category, historical dispute rate.
• Outcome streams: chargebacks with reason codes, issuer early fraud warnings (TC40 for Visa, SAFE for Mastercard), merchant-reported fraud refunds, manual review verdicts.

Label definition. Positive = a chargeback with a fraud reason code (Visa 10.x, Mastercard 4837/4863 family) or an early fraud warning or a fraud-tagged refund or a review verdict of fraud. Negative = no positive signal by the maturity date (90 days after the transaction). Non-fraud disputes ("product not received", "subscription cancelled") are not fraud; they go to a separate friendly-fraud model if needed.

Label delay. Early fraud warnings arrive in 3–10 days and cover roughly half of eventual fraud. Chargebacks peak at 30–60 days. So at time T:

• Transactions older than 90 days have mature labels.
• Transactions from 10–90 days have partial labels: positives are trustworthy, negatives are not yet.
• The last 10 days have almost no labels.

Do not treat "no chargeback yet" as negative for recent data. Either drop immature negatives or weight them by the empirical probability that a fraud would already have surfaced by that age (a survival curve learned from past data).

Selection bias. Blocked transactions never get a label. If the model is retrained only on approved traffic, it forgets the fraud it blocks and the next model is weaker on exactly those patterns. Fix: a small exploration holdout. Approve a random ~0.5% of would-be-blocked transactions with amount below a cap (e.g. $50), log the propensity, and use inverse-propensity weights in training and evaluation. Also keep 3DS-challenged outcomes; a failed challenge is a weak positive.

Class imbalance. At 0.1% positives, 6 months of data is ~30B rows and ~30M positives. Keep all positives, downsample negatives to 1% (~300M rows), and give each kept negative a weight of 100 so probabilities stay calibrated. Optionally multiply each row's weight by amount (capped) so the loss tracks dollars, not counts.

Leakage traps.

• Point-in-time correctness: every feature must be computed only from events before the authorization timestamp. Velocity counters joined by day, not by timestamp, leak the transaction itself and later attempts.
• Label-derived features: "card has a prior chargeback" must use the date the chargeback was received, not the date of the transaction it disputes.
• Merchant fraud rate must use only labels matured as of the scoring time.
• Post-auth fields (capture, refund, settlement status, issuer decline code from a later retry) must not enter features.
• Fraud rings reuse cards and devices. A random row split puts the same ring in train and test and inflates metrics.

Splits. Time-based. Train on months M−9 to M−4, validate on M−3 (labels mature by now), and backtest on several rolling windows. Also report a group-held-out split (by card fingerprint) to measure generalization to unseen cards.

Around 500–1,000 features. Group them by how fresh they must be.

Transaction (request-time, free). Amount, log amount, amount in USD, currency mismatch with card country, MCC, entry mode, 3DS/AVS/CVC results, local hour of day, card BIN attributes (issuing country, prepaid/debit/credit, card brand, issuer).

Velocity counters (streaming, seconds fresh). For keys card, device, IP, email, shipping address, BIN, merchant:

• attempts, declines, distinct merchants, total amount over 1 min / 10 min / 1 h / 24 h / 7 d
• distinct cards per device / IP / email in 1 h and 24 h (card testing shows as many cards on one device with small amounts)
• CVC or AVS failure count in the last hour

These run in a stream processor (Flink) over the auth event stream and are written to a low-latency key-value store. Card testing attacks hit in minutes, so a 1-hour batch delay is useless here.

Entity history (batch daily + streaming deltas). Card age on the platform, number of prior successful payments with this card, prior payments at this merchant, z-score of amount vs the card's usual amounts, mismatch between billing and IP country, distance between IP geolocation and shipping address, email domain age, whether email has been seen with other cards.

Cross-merchant network. The platform sees the same card at many merchants. Features: fraud rate of the card's other recent merchants, whether the card was disputed elsewhere (by receive date), number of merchants the device has paid at in 24 h. This is the main advantage a platform has over a single merchant.

Graph features (batch, hourly to daily). Build a graph of cards, devices, emails, IPs, addresses. Features: size of connected component, fraction of nodes in the component with known fraud, node2vec or GNN embedding of the card and device. Precompute and serve from the feature store.

Merchant context (batch daily). Merchant category, age, trailing 30-day matured fraud rate, typical ticket size, chargeback rate.

Feature store. One definition per feature, used both for offline point-in-time backfills and for online serving, so training and serving match. Log the exact feature vector at scoring time; train the next model on logged values where available instead of recomputed ones.

Baseline. Hand-written rules (block if > 5 cards on a device in 1 h, block on known-bad BIN ranges, step-up for first-time card over $500) plus an L2-regularized logistic regression on ~50 features. This sets the bar and stays as the fallback path.

Production model. A gradient-boosted tree ensemble (LightGBM/XGBoost), ~1,000 trees, depth 6–8.

• Handles mixed tabular features, missing values and monotone constraints (e.g. score never decreases as CVC failures increase).
• Inference is ~1–3 ms on CPU for one row.
• Loss: weighted binary cross-entropy. Weight = negative downsampling correction × capped amount weight × inverse propensity for exploration rows.

Add-ons that earn their cost.

• A sequence model (small transformer or GRU) over the card's last 50 events, trained offline, whose output embedding feeds the GBDT. Captures order patterns that counters miss (small test charge, then a large one).
• Graph embeddings from a GNN, refreshed hourly, as features.
• A separate model per segment only where traffic differs in kind (card-present vs online). Otherwise one global model with merchant features, since small merchants lack data.

Decision layer (not the model). Score → calibrated probability (isotonic regression fit on the latest matured month). Then the expected-cost rule above with merchant-specific margin and risk appetite, plus merchant-authored rules ("block if card country ≠ shipping country"). Score bands map to actions: allow, 3DS step-up for the middle band, review for merchants that opted in, block at the top. Step-up moves liability to the issuer and recovers most good customers in the gray zone.

Offline.

• PR-AUC and recall at a fixed block rate (e.g. recall when blocking 0.2% of traffic). ROC-AUC looks good at 0.1% prevalence and hides differences.
• Dollar-weighted recall at the operating point, since losses scale with amount.
• Expected cost on the backtest month, using each transaction's amount and merchant margin at the chosen threshold. This is the number that should decide model launches.
• Calibration: reliability curve and expected calibration error, per amount bucket and per large merchant. The threshold rule assumes p is a probability.
• Slices: country, card type, merchant size, new vs returning card, mobile vs web. A model that wins overall but doubles false declines in one country does not ship.
• Evaluation uses inverse-propensity weights on exploration traffic to estimate performance on traffic the old model blocked.

Online A/B.

• Randomize by card fingerprint (or by merchant for merchant-facing changes), not by request. The same fraudster then sees one arm, and per-card velocity features stay consistent.
• Fast proxy metrics (days): block rate, step-up rate and 3DS pass rate, early-fraud-warning rate on approved traffic, authorization rate, manual review approval rate.
• Slow primary metric (60–90 days): fraud loss bps and expected cost. Keep a small long-running holdback on the old model to read this.
• Guardrails: p99 latency, overall authorization rate (must not fall by more than ~0.1 pp), merchant complaints, dispute rate per merchant.

Scoring one authorization:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"sdk","x":0,"y":0,"width":220,"height":70,"label":{"text":"Checkout SDK\ndevice signals"}},
 {"type":"rectangle","id":"api","x":320,"y":0,"width":220,"height":70,"label":{"text":"Payments API"}},
 {"type":"rectangle","id":"risk","x":640,"y":0,"width":220,"height":70,"label":{"text":"Risk scoring\nservice"}},
 {"type":"rectangle","id":"gbdt","x":960,"y":0,"width":220,"height":70,"label":{"text":"GBDT model\n(in-process)"}},
 {"type":"rectangle","id":"dec","x":1280,"y":-15,"width":280,"height":100,"label":{"text":"Decision layer\ncalibration, cost rule,\nmerchant rules"}},
 {"type":"rectangle","id":"auth","x":1800,"y":-150,"width":220,"height":70,"label":{"text":"Card network\nauth"}},
 {"type":"rectangle","id":"decline","x":1800,"y":150,"width":220,"height":70,"label":{"text":"Decline to\nmerchant"}},
 {"type":"ellipse","id":"fs","x":620,"y":180,"width":260,"height":90,"label":{"text":"Online feature\nstore (Redis)"}},
 {"type":"rectangle","id":"stream","x":320,"y":380,"width":220,"height":70,"label":{"text":"Auth event\nstream"}},
 {"type":"rectangle","id":"flink","x":640,"y":380,"width":220,"height":70,"label":{"text":"Flink: velocity\ncounters"}},
 {"type":"arrow","id":"sdk-api","x":220,"y":35,"width":100,"height":0,"start":{"id":"sdk"},"end":{"id":"api"}},
 {"type":"arrow","id":"api-risk","x":540,"y":35,"width":100,"height":0,"start":{"id":"api"},"end":{"id":"risk"}},
 {"type":"arrow","id":"risk-gbdt","x":860,"y":35,"width":100,"height":0,"start":{"id":"risk"},"end":{"id":"gbdt"}},
 {"type":"arrow","id":"gbdt-dec","x":1180,"y":35,"width":100,"height":0,"start":{"id":"gbdt"},"end":{"id":"dec"}},
 {"type":"arrow","id":"dec-auth","x":1560,"y":15,"width":240,"height":-130,"start":{"id":"dec"},"end":{"id":"auth"},"label":{"text":"allow or step-up"}},
 {"type":"arrow","id":"dec-decline","x":1560,"y":55,"width":240,"height":130,"start":{"id":"dec"},"end":{"id":"decline"},"label":{"text":"block"}},
 {"type":"arrow","id":"risk-fs","x":750,"y":70,"width":1,"height":110,"start":{"id":"risk"},"end":{"id":"fs"}},
 {"type":"arrow","id":"api-stream","x":430,"y":70,"width":1,"height":310,"start":{"id":"api"},"end":{"id":"stream"}},
 {"type":"arrow","id":"stream-flink","x":540,"y":415,"width":100,"height":0,"start":{"id":"stream"},"end":{"id":"flink"}},
 {"type":"arrow","id":"flink-fs","x":750,"y":380,"width":1,"height":-110,"start":{"id":"flink"},"end":{"id":"fs"}}
]}

Labels and retraining:

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"stream","x":0,"y":0,"width":220,"height":70,"label":{"text":"Auth event\nstream"}},
 {"type":"rectangle","id":"logged","x":320,"y":0,"width":220,"height":70,"label":{"text":"Logged features\nand scores"}},
 {"type":"rectangle","id":"ts","x":640,"y":0,"width":220,"height":70,"label":{"text":"Point-in-time\ntraining set"}},
 {"type":"rectangle","id":"train","x":960,"y":0,"width":220,"height":70,"label":{"text":"Train, calibrate,\nbacktest"}},
 {"type":"rectangle","id":"gbdt","x":1440,"y":0,"width":220,"height":70,"label":{"text":"GBDT model"}},
 {"type":"rectangle","id":"cb","x":280,"y":190,"width":300,"height":90,"label":{"text":"Chargebacks, TC40\nand SAFE, reviews"}},
 {"type":"ellipse","id":"labels","x":650,"y":190,"width":200,"height":90,"label":{"text":"Label store"}},
 {"type":"arrow","id":"stream-logged","x":220,"y":35,"width":100,"height":0,"start":{"id":"stream"},"end":{"id":"logged"}},
 {"type":"arrow","id":"logged-ts","x":540,"y":35,"width":100,"height":0,"start":{"id":"logged"},"end":{"id":"ts"}},
 {"type":"arrow","id":"ts-train","x":860,"y":35,"width":100,"height":0,"start":{"id":"ts"},"end":{"id":"train"}},
 {"type":"arrow","id":"train-gbdt","x":1180,"y":35,"width":260,"height":0,"start":{"id":"train"},"end":{"id":"gbdt"},"label":{"text":"shadow then canary"}},
 {"type":"arrow","id":"cb-labels","x":580,"y":235,"width":70,"height":0,"start":{"id":"cb"},"end":{"id":"labels"}},
 {"type":"arrow","id":"labels-ts","x":750,"y":190,"width":1,"height":-120,"start":{"id":"labels"},"end":{"id":"ts"}}
]}

Latency budget (p99, 40 ms).

• Network hop and request parsing: 5 ms
• Feature fetch, ~20 keys in parallel from Redis: 10 ms
• Sequence/graph embeddings, precomputed and fetched with features: 0 extra
• GBDT inference: 3 ms
• Calibration + decision layer + merchant rules: 3 ms
• Logging (async) and headroom: 19 ms

If the feature fetch times out, score with the features present (the model was trained with random feature dropout) and mark the request; if the whole service fails, use the rules-only fallback.

Capacity. 10,000 peak QPS × 3 ms CPU ≈ 30 cores for inference alone; run ~3× that across zones for headroom and failover.

Monitoring.

• Input drift: PSI per feature, daily, against the training window; alert on PSI > 0.2 for top-50 features. Watch feature null rates (a broken SDK release silently zeroes device features).
• Score drift: block rate and score distribution per merchant, hourly. A sudden spike at one merchant is usually an attack or a merchant integration change.
• Outcome tracking: early fraud warnings per approved dollar, daily, as the earliest real signal of a miss.
• Attack response: analysts ship rules in minutes for a new pattern; the model absorbs it at the next retrain.

Feedback loops. Fraudsters adapt to whatever gets blocked, and blocked traffic loses its labels. Exploration traffic, 3DS outcomes and manual review verdicts keep labels flowing. Rules written during an attack should be reviewed and retired once the model learns the pattern, or they pile up as permanent false declines.

Retraining. Weekly retrain on a rolling 6-month window: mature data plus the most recent 10–90 days, whose negatives are weighted by the survival correction. Refit the isotonic calibrator on the latest fully matured month only; partial labels would bias probabilities low. Promotion path: offline backtest → shadow scoring for ~1 week → canary 5% → full, with automatic rollback on guardrail breach.

• GBDT over a deep model: cheap, fast, explainable with SHAP for analysts and merchants; gives up some raw sequence and graph signal, partly recovered by feeding embeddings as features.
• Expected-cost threshold vs a fixed score cutoff: adapts to amount and merchant margin; needs good calibration and per-merchant margin data that is often a guess.
• Exploration on blocked traffic: unbiased labels and honest evaluation; costs real fraud dollars (capped by amount limit and sample rate).
• Waiting for mature labels: correct negatives; the model is 3 months behind on new attack patterns. Rules and early-warning-based partial labels close the gap.
• Downsampling negatives: 100× cheaper training; noisier estimates for rare good-customer segments.
• Step-up instead of block: recovers good customers and moves liability; adds checkout friction and some abandonment (often 5–15% of challenged users drop).
• Global model with merchant features vs per-merchant models: small merchants benefit from shared signal; large merchants with unusual traffic may be under-served, so allow merchant-level thresholds and rules.
• Fail open on outage: protects revenue; a timed attack during an outage gets through, so fallback rules must still catch the obvious cases.

Mid-level (L4)

• Frames it as binary classification with heavy imbalance; uses PR-AUC, not accuracy.
• Proposes a GBDT on transaction and velocity features with a time-based split.
• Mentions that labels come from chargebacks and arrive late.

Senior (L5)

• Sets the threshold from an explicit cost model with amount and margin, and explains why calibration matters.
• Handles label maturity, point-in-time features, and the selection bias from blocked transactions (exploration with propensity weights).
• Designs the online path within a latency budget: streaming counters, feature store, fallback when features are missing.
• Picks fast proxy metrics for A/B tests and randomizes by card.

Staff (L6+)

• Treats it as an adversarial system: rules plus model, attack response in minutes, retiring stale rules.
• Uses cross-merchant network and graph signals as the platform's advantage, and handles fraud rings in evaluation splits.
• Designs the decision layer as a product: step-up vs block, liability shift, merchant-facing controls and explanations.
• Plans for failure modes: fail-open policy, SDK breakage detection, holdbacks to measure long-term loss.