Design a listing price suggestion system

• Business goal: more booked nights at healthy prices, so hosts earn more and guests find fair prices. The marketplace takes a fee on every booking, so the platform wants booking value (GBV), not the highest sticker price.
• ML task: for every (listing, night) in the open calendar, suggest a nightly price p* that maximizes expected host revenue for that night, p * P(book by night | listing, night, p, lead time), inside host bounds (min/max price) and smoothness rules. It is recomputed as the date gets closer, because demand and remaining lead time change.
• Who and when: hosts see suggestions in the calendar and can opt into automatic pricing (prices update daily without the host). New hosts see a suggestion while they create a listing.
• Scale: ~7M active listings × 365 open nights ≈ 2.5B (listing, night) cells. Full recompute daily in batch; incremental recompute for ~5% of cells when a signal changes (a booking in the area, a new event, a host changing settings).
• Latency: calendar read from a precomputed store, p99 < 50 ms. When a host changes min/max price or a discount, recompute that listing's 365 nights in < 300 ms (one batched model call, ~365 rows).
• ML objective vs business metric: the models predict booking probability and price sensitivity; the decision layer maximizes expected revenue per available night. The business metrics are host revenue, nights booked, GBV, and host adoption of suggestions. A model that maximizes revenue but that hosts ignore does nothing, so acceptance is part of the objective, not only a metric.

Two hard parts the interviewer is looking for:

1. We observe one price per night. We never see what would have happened at another price (counterfactual). Demand curves must come from price variation that exists in the data, or from controlled experiments.
2. The label is censored and delayed. An unbooked night 60 days out may still book. The final outcome is known only when the night passes.

Sources

• Listing attributes: location (lat/lng, neighborhood), room type, capacity, bedrooms, amenities, photos, house rules, minimum stay, cancellation policy.
• Calendar history: for each listing and night, the daily snapshot of price, availability, and whether it was blocked by the host. Snapshots matter; the current calendar overwrites the past.
• Bookings: booked night, booking date (gives lead time), price paid, length of stay, discounts, cancellations.
• Demand signals: searches and listing views per market, check-in date and guest count; wishlists; booking-request rejections.
• Supply: open listings per market per night; prices of comparable listings.
• External: holidays, school calendars, events (concerts, conferences, sports) with date, venue and expected size; flight search volume if licensed.
• Host behavior: whether the host accepted, edited or ignored past suggestions; opt-in to automatic pricing.

Label definition. One training row is a snapshot (listing, night, snapshot date d) with lead time L = night - d and the listed price at d. Label y = 1 if the night booked between d and the night at a price within ±5% of the snapshot price (so the price on the row is the price that sold); y = 0 if the night passed unbooked while open. Drop rows where the host blocked the night later, or where the price changed by more than 5% before it booked (the row's price is not the price that sold). Cancelled bookings count as 0 if cancelled more than 7 days before check-in, because the night returned to inventory.

• Delay: up to 365 days. Train on nights that already passed (fully resolved). For short-term freshness, add a survival-style model (below) that uses partially observed data correctly.
• Imbalance: about 30–60% of open nights book in a typical market. Far-out snapshots (L > 90) book at under 10% within the next week, so it is common to predict "books eventually" per snapshot rather than "books today".
• Sampling: 2.5B cells × daily snapshots is too much. Sample snapshots at lead-time buckets (0–3, 4–7, 8–14, 15–30, 31–60, 61–120, 121+ days), cap rows per listing, and stratify by market so large cities do not dominate. Keep weights to undo the sampling.

Leakage traps

• Using the current listing state (review count, rating, price) instead of the state as of d. All features must be point-in-time.
• Market occupancy for the target night computed after d (it contains the label).
• Prices set by the current pricing model: they create a feedback loop. Log the suggestion and whether the host used it, so you can see the policy that produced the data.
• Host blocks that are really off-platform bookings look like "unavailable" and must not be negatives.

Splits: time-based. Train on nights in months M-14 to M-3, validate on nights in M-2, test on nights in M-1. Also hold out a set of entire markets for cold-start checks. Seasonality means the validation window must cover the same season as the period you deploy into, or report per-season results.

• Comparables: a k-NN set (k ≈ 50) on embeddings of location + room type + capacity + quality, refreshed weekly. Many features are "relative to comparables", which transfers across markets with different price levels.
• Feature store: offline store with point-in-time joins for training, online store (key-value) for listing and market aggregates. Use the same feature definitions in both to avoid training/serving skew.

Baseline. Price = median nightly price of the 50 comparables for the same night type (weekday/weekend, season), times a market seasonality index from last year's same week. Cheap, explainable, and the cold-start fallback. It ignores the listing's own demand and events.

Production: three stages.

1. Booking probability model P(book | x, p, L). Gradient-boosted trees (LightGBM) with log-loss, or a survival model that predicts hazard per lead-time bucket so partially observed recent nights count (a night still open at L = 20 contributes "not booked between 60 and 20 days" without a final label). Add a monotone constraint: booking probability must not increase with price relative to comparables. This gives a sane demand curve.

2. Price elasticity. The price sensitivity in step 1 is confounded: hosts raise prices when they expect demand, so high price correlates with booking in the raw data. Remedies:

   • Use within-listing price variation over time and control for demand features (market search volume, occupancy as of d), so the model compares the same listing at similar demand.
   • Run small randomized price perturbations (±3–5%) on opted-in automatic pricing listings. These give unbiased elasticity estimates per segment; use them to calibrate or to fit the elasticity term directly.
   • Parameterize: logit P = f(x, L) - β(x) · log(p / p_ref) with β(x) > 0 learned per segment. This keeps the curve monotone and the optimum unique.

3. Price decision. For each night, search a grid of 40 prices between host min and max, compute R(p) = p · P(book | p), pick p*. Example: P(book) = 0.70 at $150 (R = $105) and 0.55 at $180 (R = $99); the suggestion is $150. Then apply rules:

   • Clip to host min/max; limit day-to-day change to ±15% unless an event is detected; smooth across adjacent nights so a weekend does not cost less than the Thursday before it without reason.
   • Last-minute discount: as L → 0 the value of waiting V_{L-1} falls toward 0 (an unsold night is worth nothing once it passes), so the optimum price falls.
   • Orphan nights (single open night between bookings) get lower prices because they book less often.

Revenue vs. occupancy. Expected revenue per night ignores that a booking blocks the night for a higher-paying later guest. A fuller formulation is dynamic pricing: value of the night at lead time L is V_L = max_p [P_L(p) · p + (1 - P_L(p)) · V_{L-1}], where P_L(p) is the chance of booking within bucket L. Approximate it with the bucketed hazard model and backward induction over 7 lead-time buckets. That is cheap per night (40 prices × 7 buckets).

Host acceptance model (optional, senior). Predict P(host accepts | suggestion, current price). Hosts reject suggestions that drop prices sharply. Optimize P(accept) · ΔR for hosts who are not on automatic pricing, which favors smaller, believable moves.

Cold start. New listings have no history. Use the comparables baseline plus quality features, with a small "new listing" discount (8–10%) to get the first reviews, which raise future demand.

Offline

• Booking model: log-loss and AUC on time-split test, by lead-time bucket and market. Calibration is required because p · P is used directly: reliability curves and expected calibration error (ECE < 0.02) per market. Recalibrate per market with isotonic regression if needed.
• Elasticity: check against randomized perturbation data. Predicted change in booking rate for a +5% price should match the measured change within its confidence interval.
• Pricing policy (counterfactual): there is no ground truth for "the best price". Use proxies on resolved nights:
  • Price-decrease recall: among nights that went unbooked, the fraction where p* < the listed price (the model would have lowered it). Target > 70%.
  • Booking regret: among booked nights, the median of max(0, price paid - p*) / price paid (money left on the table). Target < 5%.
  • These two trade against each other; report both, never one alone.
• Off-policy estimate with inverse propensity weighting on the perturbation data for expected revenue per available night.

Online A/B

• Randomize by market cluster or by listing with a holdout of whole markets, because listings in one market compete for the same guests (interference). Listing-level tests overstate gains: the treated listing steals bookings from control listings.
• Primary: host revenue per available night, nights booked, GBV.
• Secondary: suggestion adoption rate, automatic-pricing opt-in and opt-out rate.
• Guardrails: guest conversion rate (search → book), guest price-per-night satisfaction in reviews, host churn, host complaints, cancellations, price volatility per listing.
• Duration: at least 4 weeks, with readout on nights that have passed (many bookings in the test window are for future nights).

{"type":"excalidraw","elements":[
 {"type":"text","id":"title","x":0,"y":-60,"text":"Online","fontSize":20},
 {"type":"rectangle","id":"h","x":0,"y":140,"width":180,"height":70,"label":{"text":"Host calendar"}},
 {"type":"rectangle","id":"api","x":260,"y":140,"width":180,"height":70,"label":{"text":"Pricing API"}},
 {"type":"rectangle","id":"rt","x":580,"y":140,"width":220,"height":70,"label":{"text":"On-demand rescorer:\n365 nights"}},
 {"type":"ellipse","id":"kv","x":580,"y":270,"width":220,"height":90,"label":{"text":"Price store:\nlisting, night"}},
 {"type":"rectangle","id":"fs","x":900,"y":0,"width":180,"height":70,"label":{"text":"Online feature\nstore"}},
 {"type":"rectangle","id":"m","x":900,"y":140,"width":200,"height":70,"label":{"text":"Booking and\nelasticity models"}},
 {"type":"rectangle","id":"opt","x":900,"y":280,"width":180,"height":70,"label":{"text":"Optimizer +\nhost rules"}},
 {"type":"arrow","id":"h-api","x":180,"y":175,"width":80,"height":0,"start":{"id":"h"},"end":{"id":"api"}},
 {"type":"arrow","id":"api-rt","x":440,"y":175,"width":140,"height":0,"start":{"id":"api"},"end":{"id":"rt"}},
 {"type":"arrow","id":"api-kv","x":440,"y":190,"width":140,"height":125,"start":{"id":"api"},"end":{"id":"kv"}},
 {"type":"arrow","id":"rt-fs","x":800,"y":160,"width":100,"height":-125,"start":{"id":"rt"},"end":{"id":"fs"}},
 {"type":"arrow","id":"rt-m","x":800,"y":175,"width":100,"height":0,"start":{"id":"rt"},"end":{"id":"m"}},
 {"type":"arrow","id":"rt-opt","x":800,"y":190,"width":100,"height":110,"start":{"id":"rt"},"end":{"id":"opt"}},
 {"type":"arrow","id":"opt-kv","x":900,"y":330,"width":-100,"height":0,"start":{"id":"opt"},"end":{"id":"kv"}}
]}

{"type":"excalidraw","elements":[
 {"type":"text","id":"title","x":0,"y":-60,"text":"Batch","fontSize":20},
 {"type":"rectangle","id":"ev","x":0,"y":140,"width":220,"height":70,"label":{"text":"Search, bookings,\ncalendar snapshots"}},
 {"type":"rectangle","id":"str","x":300,"y":0,"width":180,"height":70,"label":{"text":"Stream\naggregates"}},
 {"type":"rectangle","id":"fs","x":580,"y":0,"width":180,"height":70,"label":{"text":"Online feature\nstore"},"strokeStyle":"dashed"},
 {"type":"ellipse","id":"lake","x":300,"y":135,"width":200,"height":80,"label":{"text":"Data lake"}},
 {"type":"rectangle","id":"train","x":580,"y":140,"width":200,"height":70,"label":{"text":"Daily training\nand calibration"}},
 {"type":"rectangle","id":"reg","x":860,"y":140,"width":180,"height":70,"label":{"text":"Model registry"}},
 {"type":"rectangle","id":"m","x":1140,"y":40,"width":200,"height":70,"label":{"text":"Booking and\nelasticity models"},"strokeStyle":"dashed"},
 {"type":"rectangle","id":"batch","x":1140,"y":250,"width":180,"height":70,"label":{"text":"Nightly batch:\n2.5B cells"}},
 {"type":"ellipse","id":"kv","x":1420,"y":240,"width":220,"height":90,"label":{"text":"Price store:\nlisting, night"},"strokeStyle":"dashed"},
 {"type":"rectangle","id":"log","x":1140,"y":400,"width":200,"height":70,"label":{"text":"Suggestion and\nacceptance logs"}},
 {"type":"arrow","id":"ev-str","x":220,"y":160,"width":80,"height":-125,"start":{"id":"ev"},"end":{"id":"str"}},
 {"type":"arrow","id":"ev-lake","x":220,"y":175,"width":80,"height":0,"start":{"id":"ev"},"end":{"id":"lake"}},
 {"type":"arrow","id":"str-fs","x":480,"y":35,"width":100,"height":0,"start":{"id":"str"},"end":{"id":"fs"}},
 {"type":"arrow","id":"lake-train","x":500,"y":175,"width":80,"height":0,"start":{"id":"lake"},"end":{"id":"train"}},
 {"type":"arrow","id":"train-reg","x":780,"y":175,"width":80,"height":0,"start":{"id":"train"},"end":{"id":"reg"}},
 {"type":"arrow","id":"reg-m","x":1040,"y":160,"width":100,"height":-85,"start":{"id":"reg"},"end":{"id":"m"}},
 {"type":"arrow","id":"reg-batch","x":1040,"y":190,"width":100,"height":95,"start":{"id":"reg"},"end":{"id":"batch"}},
 {"type":"arrow","id":"batch-kv","x":1320,"y":285,"width":100,"height":0,"start":{"id":"batch"},"end":{"id":"kv"}},
 {"type":"arrow","id":"kv-log","x":1420,"y":300,"width":-80,"height":135,"start":{"id":"kv"},"end":{"id":"log"}},
 {"type":"arrow","id":"log-lake","x":1140,"y":435,"width":-740,"height":-220,"start":{"id":"log"},"end":{"id":"lake"},"points":[[0,0],[-740,0],[-740,-220]]},
 {"type":"text","id":"legend","x":0,"y":540,"text":"Dashed boxes: online parts, from the drawing above.","fontSize":16}
]}

• Latency budget (on-demand path, 300 ms): features 60 ms, model on 365 nights × 40 prices = 14,600 rows in one batched call 150 ms, optimizer and rules 40 ms, write 30 ms, slack 20 ms.
• Batch: 2.5B cells × 40 prices = 100B model rows per day is too many. Score the full grid only for nights within 120 days; for nights further out score 10 prices around the current suggestion, and refresh them weekly.
• Monitoring: feature drift (PSI on search demand, price ratios) per market; calibration per market weekly; share of suggestions clipped by host bounds; daily distribution of suggestion changes; sudden market-wide shifts (an event the system missed shows up as fast bookings at prices far below the suggestion).
• Feedback loop: when most listings in a market use automatic pricing, the training data reflects the model's own prices and price variation shrinks. Keep the randomized perturbation slice (~2% of opted-in nights) running permanently to keep elasticity identifiable.
• Retraining: booking model weekly; calibration daily; elasticity monthly from perturbation data; market aggregates hourly. Shocks (pandemic, major disaster) need a manual override: freeze or widen bounds, and give more weight to recent weeks.

• Expected revenue vs. occupancy: maximizing p · P can leave nights empty that a lower price would fill; hosts with fixed costs (cleaning) or review goals may want occupancy. A host-chosen "aggressiveness" slider gives up one global objective for host trust.
• Observational vs. experimental elasticity: observational data is large but biased; randomized perturbations are clean but cost some host revenue and need consent for automatic-pricing users.
• Monotone parametric demand curve vs. free-form model: monotone curves give a stable, explainable optimum but can miss real non-monotone effects (very low prices can signal low quality).
• Daily batch vs. real-time: batch is cheap and stable; real-time reacts to a newly announced event in minutes but costs more and makes prices jumpy. Hybrid: batch plus event-triggered recompute.
• Market-level A/B vs. listing-level: market tests avoid interference but have far fewer units, so they need weeks and give wider confidence intervals.
• Suggesting large drops: revenue-optimal, but hosts reject them and feel the platform favors guests. Acceptance-aware optimization gives up some modeled revenue for actual adoption.

Mid-level (L4)

• Frames it as predicting booking probability at a price and picks the price that maximizes p · P(book).
• Lists sensible features (location, seasonality, comparables, lead time) and uses a time-based split.
• Proposes a comparables baseline and an A/B test on host revenue and nights booked.

Senior (L5)

• Explains the counterfactual problem and the confounding between price and demand; proposes randomized perturbations or within-listing variation.
• Handles censored, delayed labels with point-in-time snapshots and lead-time buckets; requires calibration.
• Uses offline proxies (price-decrease recall vs. booking regret) and names marketplace interference in A/B tests.

Staff (L6+)

• Treats it as dynamic pricing over lead time (value of waiting), and adds host acceptance to the objective.
• Designs for the feedback loop when most hosts use automatic pricing, plus market-level experiments and shock overrides.
• Weighs the marketplace effects: guest conversion, competition between listings, host trust, and platform GBV vs. individual host revenue.