Project deep dive: present a project you led

Question: Project deep dive: present a project you led

Companies: Anthropic, OpenAI, xAI, Harvey, Airbnb, Amazon, Applied Intuition, Figma, Microsoft, Nvidia, Waymo

Problem statement

Present a project you led in about 20 minutes: context, your role, the architecture, three hard trade-offs, metrics, cost and return, and what you would change. Then take 25 minutes of questions that go one layer deeper than your slides.

How the round works

A project deep dive is the round where your résumé gets audited. You present one project you led, usually for 15 to 25 minutes, with or without slides, and then one or two engineers spend the rest of the hour pulling on every thread. Anthropic, OpenAI, xAI and Harvey all run a version of it; at some it is a dedicated round, at others it is the first half of the hiring-manager conversation.

It tests things a whiteboard round can't:

• Depth. Do you understand the system you built down to the parts you didn't write? Can you explain why each component exists?
• Ownership. What did you decide, and what did the team or your manager decide? The interviewer will keep asking until the line is clear.
• Judgment. What were the real trade-offs, what did you give up, and would you make the same call today?
• Results. Did it work, how do you know, and what did it cost?
• Honesty. What went wrong? A project with no mistakes in it is not believable.

The common failure is a feature tour: twenty minutes of what the system does, nothing on why. Plan the talk around decisions, not components.

Choosing the project

Pick the project that best survives forty minutes of questions, not the one with the most famous name on it.

For an AI company, a project that touches models (serving, evals, data pipelines, an LLM product feature) helps, but a hard distributed-systems or reliability project told well beats an AI project told shallowly.

On confidentiality: say up front what you'll abstract ("I'll call the customer a large retailer and round the numbers"). Interviewers expect this. Never share secrets to make the story better; that's a judgment signal too.

The 20-minute talk

Use the chapters below as the outline. Rough timing:

The rest of this page works through one example project at L5/L6 scope. Remix it and replace every section with your own project; keep the shape.

Context

"I led the clause-extraction pipeline at a legal-tech company. Our customers are law firms. Before a deal closes, associates review stacks of contracts and fill in a spreadsheet: governing law, change-of-control, assignment, termination rights, about 40 clause types per contract. First-pass review took an associate around 45 minutes per contract, and our customers reviewed about 20,000 contracts a month on our platform.

We had a rules-and-regex extractor that found about 60% of clauses. Associates didn't trust it, so they re-read everything anyway. The goal was to get recall high enough that associates would review our output instead of the contract."

Two minutes, three facts: who hurts, how much, why the old way failed. Avoid company history.

My role

"I was the tech lead. I wrote the design, built the evaluation harness and the second-pass verifier myself, and led three engineers for two quarters. A product manager owned the review UI; two lawyers on our staff labelled the gold set. My manager approved the budget; the model choice and architecture were my call."

Say what you did, what others did, and who decided what. If the honest answer is "we decided together", say who argued for what.

Architecture

One diagram, explained in under four minutes: the request path first, then the one or two parts that are unusual.

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"web","x":0,"y":0,"width":180,"height":70,"label":{"text":"Web app"}},
 {"type":"rectangle","id":"api","x":340,"y":0,"width":180,"height":70,"label":{"text":"Review API"}},
 {"type":"rectangle","id":"queue","x":680,"y":0,"width":180,"height":70,"label":{"text":"Job queue"},"backgroundColor":"#ffec99","fillStyle":"solid"},
 {"type":"rectangle","id":"workers","x":680,"y":200,"width":180,"height":70,"label":{"text":"Extraction\nworkers"}},
 {"type":"rectangle","id":"llm","x":1020,"y":200,"width":180,"height":70,"label":{"text":"LLM API"}},
 {"type":"ellipse","id":"db","x":330,"y":195,"width":200,"height":80,"label":{"text":"Postgres"}},
 {"type":"rectangle","id":"eval","x":680,"y":400,"width":180,"height":70,"label":{"text":"Eval harness"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"upload","x":180,"y":35,"width":160,"height":0,"start":{"id":"web"},"end":{"id":"api"},"label":{"text":"upload"}},
 {"type":"arrow","id":"enqueue","x":520,"y":35,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"queue"},"label":{"text":"enqueue"}},
 {"type":"arrow","id":"pull","x":770,"y":70,"width":1,"height":130,"start":{"id":"queue"},"end":{"id":"workers"}},
 {"type":"arrow","id":"prompt","x":860,"y":235,"width":160,"height":0,"start":{"id":"workers"},"end":{"id":"llm"},"label":{"text":"prompts"}},
 {"type":"arrow","id":"save","x":680,"y":235,"width":-150,"height":0,"start":{"id":"workers"},"end":{"id":"db"},"label":{"text":"clauses"}},
 {"type":"arrow","id":"read","x":430,"y":70,"width":1,"height":125,"start":{"id":"api"},"end":{"id":"db"}},
 {"type":"arrow","id":"gold","x":770,"y":400,"width":1,"height":-130,"start":{"id":"eval"},"end":{"id":"workers"},"strokeStyle":"dashed"},
 {"type":"text","id":"legend","x":900,"y":420,"text":"Dashed: offline, runs\non every prompt change","fontSize":16}
]}

1. An associate uploads a contract set. The API stores the files and enqueues one job per contract.
2. Extraction workers split each contract into sections, send one prompt per contract asking for all 40 clause types as structured JSON, each with a quoted source span.
3. A verifier checks every quote against the document text. A clause whose quote doesn't appear verbatim, or that the model marked low confidence, gets a second, focused prompt with just the relevant sections (about 12% of clauses).
4. Results go to Postgres; the review UI shows each clause with its highlighted source.
5. The eval harness (dashed) runs the full gold set of 300 labelled contracts on every prompt or model change, and blocks deploys that drop recall.

The unusual part is steps 3 and 5: we treated the model as an unreliable component and built the checks around it.

Three hard trade-offs

For each: the options, the evidence, what you chose, and what it cost. This is the heart of the talk.

1. One prompt per contract, or one prompt per clause type?

• Per-clause prompts were about 2 points better on recall in our first eval, because each prompt was simpler.
• But 40 prompts × ~15,000 tokens per contract is 600,000 input tokens per contract, about 12 billion tokens a month at our volume, roughly 40× the cost and much higher latency.
• Chose: one prompt per contract for all clauses, plus a focused second pass only for low-confidence or unverified clauses. That recovered most of the recall gap (final recall within 0.5 points of per-clause) at about 2× the cost of the single pass rather than 40×.
• Gave up: a simpler pipeline; the verifier and second pass added a component we had to maintain.

2. Synchronous results or an async queue?

• Associates wanted results immediately. A 30-page contract took 40–90 seconds to process.
• Synchronous meant long-held connections, timeouts, and retries that duplicated LLM spend.
• Chose: async jobs with per-contract progress in the UI and the first results shown as soon as any contract finished. Idempotent jobs keyed by contract hash, so retries never paid twice.
• Gave up: the "instant" feel of the demo. We measured satisfaction after launch; nobody asked for synchronous.

3. Ship to a design partner now, or build the evaluation harness first?

• Product wanted a design partner using it within a month.
• I argued that without a gold set we could not tell whether a prompt change helped or hurt, and that one bad miss on a change-of-control clause would cost us the customer's trust.
• Chose: three weeks building the harness and the gold set (300 contracts, labelled by our two lawyers) before the partner launch. Agreed with the PM in advance on the launch bar: 90% recall on the 10 highest-risk clause types.
• Gave up: three weeks of calendar and about $67,500 of lawyer time (300 contracts × 1.5 hours × $150/hour). It paid back the first month: the harness caught a prompt change that improved average recall but dropped indemnification recall from 93% to 81%.

Metrics

Say how each number was measured. "Time saved" from a survey is weaker than a time study; say which one you have.

Cost and ROI

Work it through out loud; interviewers at AI companies increasingly ask what a model feature costs to run.

Running cost (LLM). Assume $3 per million input tokens and $15 per million output tokens.

• First pass: 20,000 contracts × 15,000 tokens = 300M input tokens → $900. Output about 2,000 tokens each = 40M → $600.
• Second pass: ~4.8 clauses per contract × 3,000 tokens of context = 288M input → $864; ~300 output tokens each = 28.8M → $432.
• Total about $2,800 a month, or 14 cents a contract. The per-clause design would have been about $36,000 a month in input tokens alone.

Build cost. Four engineers for two quarters ≈ 2 engineer-years, roughly $600,000 fully loaded, plus the $67,500 gold set.

Return. 30 minutes saved on 20,000 contracts = 10,000 associate hours a month. Even if only a third of that is real recovered time and we value it at $150/hour internal cost, that's about $500,000 a month for our customers. For us it showed up as the add-on that drove renewals: two of our three largest customers expanded within the year.

The lesson to say aloud: the LLM bill was not the cost that mattered. Engineering time and expert labelling dominated; the model spend was a rounding error next to the value. That changes what you optimize.

What I'd change

"Three things. First, I'd have built the eval harness even earlier, in the first week, before writing any prompt. Second, I underestimated document parsing: scanned PDFs with bad OCR caused about half our misses, and we only started fixing that in quarter two. I'd put a parsing-quality check at ingestion from day one. Third, I'd have involved the associates in labelling disagreements, not just our two lawyers; our gold set encoded two people's judgment, and some firms read clauses differently."

Good hindsight is specific, owns your own decisions, and doesn't blame other people.

Likely questions

Prepare a two-sentence answer for each. The interviewer will pick the ones that sound shakiest in your talk.

• "Why not fine-tune a model?" We had 300 labelled contracts, too few to fine-tune well, and prompts let us iterate daily. With 10× the labels I'd test a fine-tuned small model for the first pass to cut latency.
• "How did you handle a model provider outage?" Jobs stayed queued with backoff; the UI showed "processing delayed". We had a second provider configured for the first pass but never switched in production because its recall was 4 points lower; I'd have the eval decide that automatically now.
• "What happens with a 400-page contract?" Above about 100,000 tokens we split by section, ran the first pass per section and merged clauses, keeping the one with a verified quote.
• "How do you know the 95% recall holds on customer documents, not just your gold set?" We sampled 50 production contracts a month for lawyer review; production recall ran 2–3 points below the gold set, which we reported.
• "What was your hardest disagreement?" The harness-first decision with product. Explain how you agreed on the launch bar in advance.
• "What would you do with 10× the volume?" Batch API for non-urgent jobs (cheaper), cache by contract hash (firms re-upload the same templates), and per-customer rate limits on the queue.
• "What part did you not understand well?" Have an honest answer: for example, the review UI's highlighting code, or the OCR vendor's internals.
• "If you started over today, would you build this?" Have a view, including whether newer models make parts of it unnecessary.

Expect at least one question that goes one layer deeper than your diagram (how the queue retries, how the verifier normalizes whitespace in quotes). If you don't know, say what you'd check.

Interviewer expectations

Mid-level (L4)

• Presents a project they built a clear part of, in a logical order, with one diagram.
• Explains why each component exists and answers implementation questions on their own part in depth.
• Has before/after numbers and one honest thing that went wrong.

Senior (L5)

• Organizes the talk around decisions: at least three trade-offs with options, evidence, choice and cost.
• Separates their contribution from the team's without being asked.
• Knows the metrics, how they were measured, and the running cost; answers "what would you change" with specifics.
• Handles a question one layer below their diagram, or says exactly how they'd find out.

Staff (L6+)

• The project changed how a team or organization works (an eval standard, a platform, a cost model), not just a feature.
• Reasons about ROI: build cost, running cost, value, and which of those actually mattered, with numbers.
• Discusses the paths not taken and the conditions under which they'd be right; revisits their own calls in light of what changed since (newer models, more data).
• Shows how they handled disagreement with product or leadership and set decision rules in advance.