Design a family-friendly listing ranking system

Business goal. Guests who search with children (the search form has children and infant counts) should book faster, have a stay that works for kids, and come back. A bad family stay (unfenced pool, no crib, steep open stairs, a party street) costs a refund, a support ticket and often the guest.

ML task. For a search with children + infants > 0, re-rank the candidate listings by expected value for this family:
score = P(book | query, listing, family context) x E[good family stay | booked].
There is no ground-truth "family-friendly" label, so the core sub-problem is building a listing-level family suitability score from weak signals, and then using it, plus outcome labels from real family stays, inside the ranker.

Scope and scale (assumptions to state out loud).

• ~7M active listings; ~20% of searches carry children or infants; peak search ~20k QPS, so ~4k QPS of family searches.
• The geo/date/capacity retrieval returns ~1-2k available candidates; we rank and show the top ~20-50.
• Ranking budget: ~60 ms p99 inside a ~300 ms search request.
• Only applied when the family context is explicit. Many families leave the child count at 0; inferring family intent is a v2 (see Trade-offs).

ML objective vs business metric. Offline we optimize ranking loss on booked listings in family sessions plus a calibrated stay-quality head. The business metrics are family booking conversion, family stay rating, child-related support tickets and refunds, and 12-month rebooking rate for family guests. Booking alone is the wrong target: a listing with great photos and a balcony with no railing converts well and reviews badly.

Sources.

• Listing content: structured amenities (crib, high chair, travel cot, stair gates, children's books/toys, fenced pool, baby bath), property type, bedroom/bed layout, house rules ("not suitable for children 2-12 / infants"), description text, photos.
• Booking logs: guest counts by age band, trip dates, cancellations.
• Reviews: text and sub-ratings, plus whether the stay had children.
• Support: tickets and refunds tagged with safety or "listing not as described".
• Search logs: impressions, positions, clicks, wishlists, bookings, with the query's family context.

Three label tiers, from weak to strong.

1. Weak suitability labels (every listing). Labeling functions (LFs), each voting +1 / -1 / abstain:
   • +1: has crib or travel cot and high chair; host lists child-safety amenities; >= 2 separate bedrooms for 2 adults + 2 kids.
   • +1: reviews from family stays mention kids positively ("kids loved the garden"), via a text classifier.
   • +1: family booking rate given family impressions is above market median (exposure-normalized, not raw counts).
   • -1: host marked "not suitable for children/infants" (also a hard filter).
   • -1: unfenced pool or hot tub, open lofts, "party" or nightlife keywords, studio with one bed for 4 guests.
   • -1: reviews or tickets from family stays mention safety problems.
     A label model (Snorkel-style generative model) learns each LF's accuracy and correlation from their agreement and emits a probabilistic label p_family per listing.
2. Golden set (~5k listings). Trained raters score listings 1-5 against a rubric (safety, space, kid amenities, neighborhood), stratified by market, property type and booking volume. Two raters per listing; keep listings with agreement (target Cohen's kappa >= 0.6). Used to measure LF accuracy, tune the label model, and as the offline test set. Never used for training.
3. Outcome labels (family bookings only). good_family_stay = 1 if the stay was not cancelled by the guest after check-in, overall rating >= 4.5 (or no review and no ticket), no child- or safety-related ticket or refund. Label delay = lead time + stay + 14-day review window, typically 30-90 days.

Ranking labels. Family search sessions: booked listing = positive (graded: booked > wishlisted > clicked > impressed). ~1-2% of family sessions end in a booking, so sample negatives within the same session (impressed, not clicked), not across sessions.

Imbalance. Clear positives in tier 1 are maybe 10-15% of listings; bookings are rare per impression; bad family stays are ~3-5% of family bookings. Use per-session sampling for ranking and class weights or focal loss for the bad-stay head.

Leakage traps.

• Point-in-time features: "past family bookings", review-derived scores and amenity lists must be computed as of the search time, not today. A host who adds "crib" after a great family stay must not leak into the training row for that stay.
• An LF that uses a signal must not also be a raw feature of the end model that learns from that LF, or the model just memorizes the LF. The end suitability model uses content (text, photos, amenities, layout); behavioral signals stay in the LFs and the ranker.
• The target booking's own review must not appear in its review features.

Splits. Time-based: train on months 1-10, validate on 11, test on 12, with outcome labels matured (drop bookings whose review window is still open). The suitability model also gets a listing-held-out split (new listings in the test months) to measure cold start.

Selection bias. Families book what they were shown, and today's ranker already hides some listings. Log propensities (position, plus a small randomized exploration slot) and use inverse propensity weighting for the stay-quality head, which is trained only on booked stays.

The feature store holds point-in-time offline tables for training and a low-latency online store (key = listing_id) for serving. ~7M listings x (64 floats x 4 B + ~100 scalars) is ~4.6 GB: fits in memory across a few shards.

Baseline (ship in weeks). Existing search ranker, plus: hard-filter listings marked not suitable for the given ages; a hand-weighted boost for kid amenities matched to the query (crib when infants > 0). Gives a measurable floor and the logs for v1.

Stage 1: family suitability model (offline, per listing).

• Train on p_family soft labels from the label model with cross-entropy against soft targets.
• Inputs: amenity flags, layout, text embedding of description and house rules (small fine-tuned text encoder), photo detections from a vision model, neighborhood features.
• Model: GBDT on the tabular plus embedding features first; a multimodal network later if photos carry lift.
• Output: calibrated family_score in [0,1] and a 64-d embedding. Because it reads content, it scores brand-new listings with zero bookings.
• An LLM can extract structured attributes from descriptions and reviews ("gated pool", "steep spiral stairs") to feed both LFs and features; run it offline, never in the request path.

Stage 2: family-aware ranker (online).

• Retrieval unchanged (geo, dates, capacity, hard filters).
• Ranker: the main search DNN or GBDT ranker, with the family features and crosses above, trained on all sessions with the family context as an input, so it learns when family features matter.
• Multi-task heads:
  • Booking head: listwise softmax loss over each session's impressed listings (or LambdaRank on NDCG).
  • Stay-quality head: binary cross-entropy on good_family_stay, family bookings only, IPW-weighted, with label smoothing toward the family_score prior for listings with few family stays.
• Final score: log P(book) + lambda * log P(good stay), with lambda tuned by online experiments (start ~0.5). This keeps conversion as the main signal and stops a listing that converts but disappoints families from winning.
• Light diversity re-rank in the top 20 (property types, price bands) so the page is not all large villas.

Suitability model (offline).

• On the golden set: AUC and PR-AUC for rubric score >= 4, Spearman correlation with the 1-5 score; slices by market, property type, new vs established listings.
• Label model sanity check: per-LF accuracy and coverage against the golden set; drop LFs below ~60% accuracy.
• Calibration: reliability curve and ECE of family_score on the golden set (it is used as a prior and a feature, so scale matters).

Ranker (offline).

• NDCG@10 and MRR of the booked listing in family sessions from the test month, vs the production ranker.
• Stay-quality head: AUC, and ECE on matured family bookings, because it multiplies the booking probability.
• "Bad-stay exposure": share of top-10 slots taken by listings with a known bad family outcome or a negative rubric score. Must go down.
• Counterfactual check with IPS on the exploration traffic before any online test.

Online A/B (family searches only, randomized by guest).

• Primary: family booking conversion per search.
• Secondary (lagging, 60-90 days): family stay rating, child or safety-related tickets and refunds per 1k family stays, family-guest rebooking.
• Guardrails: overall bookings and revenue; non-family search metrics (must be flat, since treatment only applies when children > 0); p99 latency; booking concentration across hosts (Gini) and new-listing share of family bookings; cancellation rate.
• Because stay outcomes lag, decide on conversion plus early proxies (wishlist rate, host messages asking "is it child-safe?"), and keep a long-running holdback (~2%) to read the lagging metrics.

{"type":"excalidraw","elements":[
 {"type":"text","id":"title","x":0,"y":-50,"text":"Online","fontSize":20},
 {"type":"rectangle","id":"q","x":0,"y":0,"width":220,"height":100,"label":{"text":"Search request:\ndest, dates, adults,\nchildren, infants"}},
 {"type":"rectangle","id":"r","x":300,"y":0,"width":240,"height":100,"label":{"text":"Retrieval: geo,\navailability, capacity,\nhard filters"}},
 {"type":"rectangle","id":"f","x":620,"y":15,"width":180,"height":70,"label":{"text":"Feature fetch:\nonline store"}},
 {"type":"rectangle","id":"m","x":600,"y":180,"width":220,"height":100,"label":{"text":"Family-aware ranker:\nbook head +\nstay head"}},
 {"type":"rectangle","id":"d","x":280,"y":180,"width":240,"height":100,"label":{"text":"Diversity re-rank\nand exploration slot"}},
 {"type":"rectangle","id":"res","x":0,"y":195,"width":180,"height":70,"label":{"text":"Results page"}},
 {"type":"arrow","id":"q-r","x":220,"y":50,"width":80,"height":0,"start":{"id":"q"},"end":{"id":"r"}},
 {"type":"arrow","id":"r-f","x":540,"y":50,"width":80,"height":0,"start":{"id":"r"},"end":{"id":"f"}},
 {"type":"arrow","id":"f-m","x":710,"y":85,"width":1,"height":95,"start":{"id":"f"},"end":{"id":"m"}},
 {"type":"arrow","id":"m-d","x":600,"y":230,"width":-80,"height":0,"start":{"id":"m"},"end":{"id":"d"}},
 {"type":"arrow","id":"d-res","x":280,"y":230,"width":-100,"height":0,"start":{"id":"d"},"end":{"id":"res"}}
]}

{"type":"excalidraw","elements":[
 {"type":"text","id":"title","x":0,"y":-160,"text":"Offline","fontSize":20},
 {"type":"rectangle","id":"l","x":0,"y":0,"width":200,"height":70,"label":{"text":"Listings, photos,\nreviews, tickets"}},
 {"type":"rectangle","id":"lf","x":280,"y":0,"width":180,"height":70,"label":{"text":"Labeling\nfunctions"}},
 {"type":"rectangle","id":"lm","x":540,"y":0,"width":180,"height":70,"label":{"text":"Label model"}},
 {"type":"rectangle","id":"g","x":540,"y":-160,"width":180,"height":70,"label":{"text":"Golden set,\nrater rubric"}},
 {"type":"rectangle","id":"s","x":800,"y":-15,"width":220,"height":100,"label":{"text":"Suitability model:\nfamily_score,\nembedding"}},
 {"type":"ellipse","id":"fs","x":1090,"y":-5,"width":200,"height":80,"label":{"text":"Feature store"}},
 {"type":"rectangle","id":"f","x":1370,"y":0,"width":180,"height":70,"label":{"text":"Feature fetch:\nonline store"},"strokeStyle":"dashed"},
 {"type":"rectangle","id":"res","x":540,"y":200,"width":180,"height":70,"label":{"text":"Results page"},"strokeStyle":"dashed"},
 {"type":"rectangle","id":"log","x":800,"y":200,"width":220,"height":70,"label":{"text":"Impression, booking\nand stay logs"}},
 {"type":"rectangle","id":"t","x":1100,"y":200,"width":180,"height":70,"label":{"text":"Ranker training"}},
 {"type":"rectangle","id":"m","x":1400,"y":185,"width":220,"height":100,"label":{"text":"Family-aware ranker:\nbook head +\nstay head"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"l-lf","x":200,"y":35,"width":80,"height":0,"start":{"id":"l"},"end":{"id":"lf"}},
 {"type":"arrow","id":"lf-lm","x":460,"y":35,"width":80,"height":0,"start":{"id":"lf"},"end":{"id":"lm"}},
 {"type":"arrow","id":"g-lm","x":630,"y":-90,"width":1,"height":90,"start":{"id":"g"},"end":{"id":"lm"}},
 {"type":"arrow","id":"lm-s","x":720,"y":35,"width":80,"height":0,"start":{"id":"lm"},"end":{"id":"s"}},
 {"type":"arrow","id":"s-fs","x":1020,"y":35,"width":70,"height":0,"start":{"id":"s"},"end":{"id":"fs"}},
 {"type":"arrow","id":"fs-f","x":1290,"y":35,"width":80,"height":0,"start":{"id":"fs"},"end":{"id":"f"}},
 {"type":"arrow","id":"fs-t","x":1190,"y":75,"width":1,"height":125,"start":{"id":"fs"},"end":{"id":"t"}},
 {"type":"arrow","id":"res-log","x":720,"y":235,"width":80,"height":0,"start":{"id":"res"},"end":{"id":"log"}},
 {"type":"arrow","id":"log-t","x":1020,"y":235,"width":80,"height":0,"start":{"id":"log"},"end":{"id":"t"}},
 {"type":"arrow","id":"t-m","x":1280,"y":235,"width":120,"height":0,"start":{"id":"t"},"end":{"id":"m"}},
 {"type":"text","id":"legend","x":0,"y":330,"text":"Dashed boxes: online parts, from the drawing above.","fontSize":16}
]}

Latency budget (ranking part, p99). Feature fetch for ~1.5k candidates 20 ms (batched, sharded store); ranker inference 30 ms (GBDT or small DNN on CPU, candidates scored in batches); re-rank and exploration 5 ms; slack 5 ms. Total ~60 ms. Everything expensive (vision, text encoder, LLM extraction, label model) runs offline.

Cadence.

• Suitability scores: daily batch; listing edits and new photos trigger a re-score within ~1 h through a stream job.
• Ranker: retrain weekly on a rolling 12-month window; stay-quality head monthly as outcome labels mature.
• Label model and LFs: review quarterly against a refreshed golden sample (~500 new rated listings per quarter).

Monitoring.

• Feature drift (PSI) on family_score and key amenity flags per market; spikes in amenity edits (host gaming: everyone adds "crib").
• Prediction drift: share of top-10 listings with family_score > 0.7.
• Calibration of the stay head on newly matured bookings, by market.
• Business: daily family conversion and weekly child-related ticket rate, with alerts.

Feedback loops. Boosted listings get more family bookings, which raises their behavioral LF and their training positives. Counter it with exposure-normalized behavioral signals (family bookings per family impression), a 1-2% exploration slot for listings with high content score but little family history, and IPW in training. Amenity gaming is checked against photo detections and family-stay reviews; a mismatch lowers the listing's trust weight and sends it to review.

• Weak supervision vs paying raters for training labels. LFs cover all 7M listings cheaply and adapt by editing rules, but they carry systematic errors (e.g., they over-reward large homes). Raters are accurate but cover ~5k listings, so we spend them on evaluation, not training.
• Content-only suitability model vs using behavior. Content generalizes to new listings and avoids the feedback loop, but misses what only families who stayed know (noise, real crib quality). Behavior goes to the ranker, where it is exposure-normalized.
• Score multiplication with lambda vs a single booking objective. Protects stay quality, but costs some short-term conversion and needs long holdouts to tune.
• Hard filter on "not suitable for children". Respects host rules and avoids bad stays, but removes supply in tight markets. Show a count of filtered listings rather than silently shrinking results.
• Explicit family context only. Precise and easy to explain, but misses families who don't enter children. A family-intent classifier (past trips with kids, 2 adults + 3 beds, filter use) would raise coverage at the risk of wrongly re-ranking for groups of friends; ship it as a soft feature, not a switch.
• Daily batch scores. Cheap and stable, but a listing that adds a crib waits up to a day unless the edit stream job catches it.
• Exploration slot. Better data on new listings, at a small conversion cost (~0.1-0.3% in exposed sessions).

Mid-level (L4)

• Uses the children/infant count as the trigger and proposes amenities plus a hard filter as a baseline.
• Builds a proxy label from bookings with children and trains a classifier; time-based split.
• Measures NDCG offline and family conversion in an A/B test.

Senior (L5)

• Separates listing suitability (weak labels, content model, cold start) from the ranker, and explains why booking is not the right sole label.
• Designs labeling functions plus a label model, a golden set used only for evaluation, and point-in-time features to stop leakage.
• Adds a stay-quality head with delayed labels and IPW, calibration, lagging metrics with a holdback, and guardrails for non-family searches.

Staff (L6+)

• Names the feedback loop and host gaming, and designs exposure normalization, exploration and trust checks against them.
• Frames the policy choices: hard filter vs supply loss, inferring family intent, how lambda trades conversion against stay quality, and who owns that trade.
• Plans the system's evolution: refresh of the label model and golden set, reuse of the suitability pipeline for other weakly labeled attributes (pet-friendly, accessible, remote-work-ready).