0

Design an answer engine with citations

by Frontier Interview· about 4 hours ago· ⑂ 0 remixes

Question: Design an answer engine with citations

Companies: Perplexity

Difficulty: Hard

Problem statement

Design an answer engine: a user asks a question in plain language and gets a direct, current answer written by a model, with numbered citations to the web pages it used, streaming within a second or two, with follow-up questions in the same conversation. Cover query rewriting, search and page fetching, passage selection, generation, how you make sure each citation actually supports its sentence, freshness, caching, a latency budget per stage, and cost at 30 million queries a day.

Requirements

An answer engine: the user asks a question in plain language and gets a direct, current answer written by a model, with numbered citations to the web pages it came from, streamed within a second or two. Follow-up questions keep the conversation's context.

Functional requirements

  • Answer any question with a synthesized answer and inline citations [1], [2] that link to the sources.
  • Use fresh information: news from the last hour should be findable.
  • Support follow-ups ("and what about in Europe?") that reuse the conversation.
  • Show the sources list, and let users open each one.

Non-functional requirements

  • First answer token in under 1.5 s at p50, under 3 s at p95; full answer in about 5 to 8 s.
  • Every factual sentence should be supported by a cited source; citations must point at text that supports the sentence.
  • Scale: 30M queries a day (about 350 a second on average, about 1,000 at peak).
  • Cost per answer low enough for a free tier: cents, not dimes.

Back-of-the-envelope estimation

  • Latency budget for first token (1.5 s): query understanding 100 ms, search 250 ms, fetching and extracting the top pages 500 ms (parallel, hard timeout), reranking 100 ms, model time to first token 400 ms, plus network: about 1.4 s. Every stage needs a timeout and a fallback.
  • Tokens: about 10 passages × 400 tokens + 1,000 tokens of instructions and history = about 5,000 input tokens, 400 output tokens. At 30M answers a day that is 150B input and 12B output tokens a day. Serving it on owned GPUs with a fine-tuned open model for most queries and a frontier model for hard ones keeps the cost to a few tenths of a cent per answer.
  • Fetching: 10 pages per query × 1,000 queries a second = 10,000 page fetches a second at peak. Most must come from a page cache (popular pages are asked about again and again); a 70% cache hit rate leaves 3,000 live fetches a second.
  • Index: a web index of a few billion pages for retrieval (keyword plus embeddings of passages), plus a fresh tier for news updated every few minutes.

API design

  • POST /v1/answers {question, conversation_id?, focus?} → server-sent events: {status: "searching"}, {sources: [...]}, {delta: "..."}, {citations_final, related_questions}
  • GET /v1/conversations/{id} → turns with answers and sources

Data model

  • conversations(id, user_id, created_at), turns(conversation_id, seq, question, rewritten_query, answer, sources[], model, latency_ms)
  • pages(url, fetched_at, content_hash, extracted_text_ref, title, published_at): the page cache.
  • Search index: passages (url, passage_id, text, embedding, published_at, domain_quality).
  • answer_cache(normalized_query, freshness_class) -> answer, sources, expires_at for popular evergreen questions.

High-level design

Build it one requirement at a time. Each drawing adds to the one before; the shaded boxes are what that step adds.

1) Understand the question and search

The answer API streams results back. A query planner rewrites the question into one or more search queries (resolving follow-ups against the conversation: "and in Europe?" becomes "EV sales Europe 2026"), decides freshness needs, and searches the web index.

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"user","x":0,"y":0,"width":180,"height":70,"label":{"text":"User"}},
 {"type":"rectangle","id":"api","x":340,"y":0,"width":180,"height":70,"label":{"text":"Answer API"}},
 {"type":"rectangle","id":"plan","x":680,"y":0,"width":180,"height":70,"label":{"text":"Query planner"}},
 {"type":"rectangle","id":"search","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Search"}},
 {"type":"ellipse","id":"index","x":1000,"y":155,"width":220,"height":80,"label":{"text":"Web index"}},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"user"},"end":{"id":"api"},"label":{"text":"question"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"plan"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"plan"},"end":{"id":"search"},"label":{"text":"queries"}},
 {"type":"arrow","id":"a4","x":1110,"y":70,"width":1,"height":85,"start":{"id":"search"},"end":{"id":"index"}}
]}

2) Read the best pages

Search returns about 30 candidate URLs with snippets. A fetcher pulls the top 10 to 15 pages in parallel (from the page cache when fresh enough), extracts the main text, and splits it into passages. A reranker scores passages against the question and keeps about 10.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-80,"text":"Shaded: added in this step","fontSize":16},
 {"type":"rectangle","id":"user","x":0,"y":0,"width":180,"height":70,"label":{"text":"User"}},
 {"type":"rectangle","id":"api","x":340,"y":0,"width":180,"height":70,"label":{"text":"Answer API"}},
 {"type":"rectangle","id":"plan","x":680,"y":0,"width":180,"height":70,"label":{"text":"Query planner"}},
 {"type":"rectangle","id":"search","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Search"}},
 {"type":"ellipse","id":"index","x":1000,"y":155,"width":220,"height":80,"label":{"text":"Web index"}},
 {"type":"rectangle","id":"fetch","x":1360,"y":0,"width":180,"height":70,"label":{"text":"Fetch + extract"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"ellipse","id":"pages","x":1340,"y":155,"width":220,"height":80,"label":{"text":"Page cache"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"rerank","x":1360,"y":320,"width":180,"height":70,"label":{"text":"Reranker"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"user"},"end":{"id":"api"},"label":{"text":"question"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"plan"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"plan"},"end":{"id":"search"},"label":{"text":"queries"}},
 {"type":"arrow","id":"a4","x":1110,"y":70,"width":1,"height":85,"start":{"id":"search"},"end":{"id":"index"}},
 {"type":"arrow","id":"a5","x":1200,"y":35,"width":160,"height":0,"start":{"id":"search"},"end":{"id":"fetch"},"label":{"text":"URLs"}},
 {"type":"arrow","id":"a6","x":1450,"y":70,"width":1,"height":85,"start":{"id":"fetch"},"end":{"id":"pages"}},
 {"type":"arrow","id":"a7","x":1580,"y":35,"width":0,"height":320,"start":{"id":"fetch"},"end":{"id":"rerank"},"points":[[-40,0],[40,0],[40,320],[-40,320]]}
]}

3) Write the answer with checked citations

The model gets the question, conversation summary and numbered passages, and streams an answer that cites passage numbers. A citation checker runs on each finished sentence: it drops citations whose passage does not support the sentence, merges passages from the same URL into one number, renumbers by first use, and flags uncited claims. Popular evergreen answers are cached.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-80,"text":"Shaded: added in this step. Dashed: the streamed answer.","fontSize":16},
 {"type":"rectangle","id":"user","x":0,"y":0,"width":180,"height":70,"label":{"text":"User"}},
 {"type":"rectangle","id":"api","x":340,"y":0,"width":180,"height":70,"label":{"text":"Answer API"}},
 {"type":"ellipse","id":"cache","x":320,"y":155,"width":220,"height":80,"label":{"text":"Answer cache"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"plan","x":680,"y":0,"width":180,"height":70,"label":{"text":"Query planner"}},
 {"type":"rectangle","id":"search","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Search"}},
 {"type":"ellipse","id":"index","x":1000,"y":155,"width":220,"height":80,"label":{"text":"Web index"}},
 {"type":"rectangle","id":"fetch","x":1360,"y":0,"width":180,"height":70,"label":{"text":"Fetch + extract"}},
 {"type":"ellipse","id":"pages","x":1340,"y":155,"width":220,"height":80,"label":{"text":"Page cache"}},
 {"type":"rectangle","id":"rerank","x":1360,"y":320,"width":180,"height":70,"label":{"text":"Reranker"}},
 {"type":"rectangle","id":"llm","x":1020,"y":320,"width":180,"height":70,"label":{"text":"Answer model"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"check","x":680,"y":320,"width":180,"height":70,"label":{"text":"Citation check"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"user"},"end":{"id":"api"},"label":{"text":"question"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"plan"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"plan"},"end":{"id":"search"},"label":{"text":"queries"}},
 {"type":"arrow","id":"a4","x":1110,"y":70,"width":1,"height":85,"start":{"id":"search"},"end":{"id":"index"}},
 {"type":"arrow","id":"a5","x":1200,"y":35,"width":160,"height":0,"start":{"id":"search"},"end":{"id":"fetch"},"label":{"text":"URLs"}},
 {"type":"arrow","id":"a6","x":1450,"y":70,"width":1,"height":85,"start":{"id":"fetch"},"end":{"id":"pages"}},
 {"type":"arrow","id":"a7","x":1580,"y":35,"width":0,"height":320,"start":{"id":"fetch"},"end":{"id":"rerank"},"points":[[-40,0],[40,0],[40,320],[-40,320]]},
 {"type":"arrow","id":"a8","x":1360,"y":355,"width":-160,"height":0,"start":{"id":"rerank"},"end":{"id":"llm"},"label":{"text":"passages"}},
 {"type":"arrow","id":"a9","x":1020,"y":355,"width":-160,"height":0,"start":{"id":"llm"},"end":{"id":"check"},"label":{"text":"tokens"}},
 {"type":"arrow","id":"a10","x":690,"y":320,"width":-200,"height":-250,"start":{"id":"check"},"end":{"id":"api"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"a11","x":430,"y":70,"width":1,"height":85,"start":{"id":"api"},"end":{"id":"cache"}}
]}

Final design

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-80,"text":"Dashed: the streamed answer.","fontSize":16},
 {"type":"rectangle","id":"user","x":0,"y":0,"width":180,"height":70,"label":{"text":"User"}},
 {"type":"rectangle","id":"api","x":340,"y":0,"width":180,"height":70,"label":{"text":"Answer API"}},
 {"type":"ellipse","id":"cache","x":320,"y":155,"width":220,"height":80,"label":{"text":"Answer cache"}},
 {"type":"rectangle","id":"plan","x":680,"y":0,"width":180,"height":70,"label":{"text":"Query planner"}},
 {"type":"rectangle","id":"search","x":1020,"y":0,"width":180,"height":70,"label":{"text":"Search"}},
 {"type":"ellipse","id":"index","x":1000,"y":155,"width":220,"height":80,"label":{"text":"Web index"}},
 {"type":"rectangle","id":"fetch","x":1360,"y":0,"width":180,"height":70,"label":{"text":"Fetch + extract"}},
 {"type":"ellipse","id":"pages","x":1340,"y":155,"width":220,"height":80,"label":{"text":"Page cache"}},
 {"type":"rectangle","id":"rerank","x":1360,"y":320,"width":180,"height":70,"label":{"text":"Reranker"}},
 {"type":"rectangle","id":"llm","x":1020,"y":320,"width":180,"height":70,"label":{"text":"Answer model"}},
 {"type":"rectangle","id":"check","x":680,"y":320,"width":180,"height":70,"label":{"text":"Citation check"}},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"user"},"end":{"id":"api"},"label":{"text":"question"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"plan"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"plan"},"end":{"id":"search"},"label":{"text":"queries"}},
 {"type":"arrow","id":"a4","x":1110,"y":70,"width":1,"height":85,"start":{"id":"search"},"end":{"id":"index"}},
 {"type":"arrow","id":"a5","x":1200,"y":35,"width":160,"height":0,"start":{"id":"search"},"end":{"id":"fetch"},"label":{"text":"URLs"}},
 {"type":"arrow","id":"a6","x":1450,"y":70,"width":1,"height":85,"start":{"id":"fetch"},"end":{"id":"pages"}},
 {"type":"arrow","id":"a7","x":1580,"y":35,"width":0,"height":320,"start":{"id":"fetch"},"end":{"id":"rerank"},"points":[[-40,0],[40,0],[40,320],[-40,320]]},
 {"type":"arrow","id":"a8","x":1360,"y":355,"width":-160,"height":0,"start":{"id":"rerank"},"end":{"id":"llm"},"label":{"text":"passages"}},
 {"type":"arrow","id":"a9","x":1020,"y":355,"width":-160,"height":0,"start":{"id":"llm"},"end":{"id":"check"},"label":{"text":"tokens"}},
 {"type":"arrow","id":"a10","x":690,"y":320,"width":-200,"height":-250,"start":{"id":"check"},"end":{"id":"api"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"a11","x":430,"y":70,"width":1,"height":85,"start":{"id":"api"},"end":{"id":"cache"}}
]}

Deep dives

Query planning

A small, fast model (or a fine-tuned classifier plus rewriter) does four things in about 100 ms:

  • Rewrite follow-ups into standalone queries using the conversation.
  • Split compound questions ("compare the iPhone 17 and Pixel 11 cameras") into parallel searches.
  • Classify freshness: news and prices need results from the last hours; evergreen questions do not. This picks the index tier and the cache policy.
  • Pick the answer model: simple lookups go to a small model; multi-step reasoning or comparisons to a larger one.

Retrieval and fetching

Search runs hybrid retrieval (keyword plus passage embeddings) over the main index and the fresh tier, then boosts by domain quality and recency. Snippets from the index are often enough for simple questions; fetching full pages improves answers that need details, so fetch the top 10 to 15 in parallel with a hard deadline (say 600 ms) and use whatever arrived. Extraction strips navigation, ads and boilerplate; pages that block bots or need JavaScript fall back to the indexed snippet.

The reranker (a cross-encoder) scores about 100 passages against the question and keeps about 10, with a cap per domain so one site does not dominate the sources.

Generation and citations

The prompt numbers each passage and instructs the model to answer only from them, cite the passage numbers after each claim, and say when sources disagree or do not answer. The model will still sometimes cite a passage that does not support the sentence, or cite a number it was not given. The citation checker runs on each completed sentence while the answer streams:

import re
from typing import Dict, List, Tuple

MARK = re.compile(r"\[(\d+)\]")
WORD = re.compile(r"[a-z0-9]+")
STOP = {"the", "a", "an", "of", "and", "or", "to", "in", "is", "are", "was", "for", "on", "with", "by", "it", "that"}


def _words(text: str) -> set:
    return {w for w in WORD.findall(text.lower()) if w not in STOP}


def support(sentence: str, passage: str) -> float:
    """Share of the sentence's content words that appear in the passage."""
    words = _words(MARK.sub("", sentence))
    return len(words & _words(passage)) / len(words) if words else 0.0


def finalize_citations(answer: str, passages: Dict[int, str], urls: Dict[int, str],
                       min_support: float = 0.3) -> Tuple[str, List[str], List[str]]:
    """Check and renumber the model's [n] markers.

    - drops markers that point at passages we never gave the model;
    - drops markers whose passage does not support the sentence;
    - renumbers the rest 1, 2, 3 in order of first use, one number per URL;
    - reports sentences that make claims with no surviving citation.
    Returns (answer, sources in display order, unsupported sentences).
    """
    sentences = re.split(r"(?<=[.!?])\s+", answer.strip())
    number_for_url: Dict[str, int] = {}
    sources: List[str] = []
    out_sentences, unsupported = [], []
    for sentence in sentences:
        kept: List[int] = []
        for n in (int(m) for m in MARK.findall(sentence)):
            if n in passages and support(sentence, passages[n]) >= min_support:
                url = urls[n]
                if url not in number_for_url:
                    number_for_url[url] = len(sources) + 1
                    sources.append(url)
                if number_for_url[url] not in kept:
                    kept.append(number_for_url[url])
        body = MARK.sub("", sentence).rstrip()
        body = re.sub(r"\s+([.!?,;:])", r"\1", body)
        if kept:
            marks = "".join(f"[{k}]" for k in kept)
            body = body[:-1] + marks + body[-1] if body and body[-1] in ".!?" else body + marks
        elif len(_words(body)) >= 4:
            unsupported.append(body)
        out_sentences.append(body)
    return " ".join(out_sentences), sources, unsupported

_P = {
    1: "The Eiffel Tower is 330 metres tall since a 2022 antenna was added.",
    2: "The Louvre in Paris is the most visited museum in the world.",
    3: "Gustave Eiffel's company built the tower for the 1889 World Fair.",
}
_U = {1: "https://a.example/tower", 2: "https://b.example/louvre", 3: "https://a.example/tower"}


def test_citations_are_checked_merged_and_renumbered():
    answer = ("The Eiffel Tower is 330 metres tall [1][7]. It was built for the 1889 World Fair [3]. "
              "The Louvre is the most visited museum [2][1]. It is repainted every seven years with sixty tonnes of paint.")
    text, sources, unsupported = finalize_citations(answer, _P, _U)
    assert text == ("The Eiffel Tower is 330 metres tall[1]. It was built for the 1889 World Fair[1]. "
                    "The Louvre is the most visited museum[2]. It is repainted every seven years with sixty tonnes of paint.")
    assert sources == ["https://a.example/tower", "https://b.example/louvre"]
    assert unsupported == ["It is repainted every seven years with sixty tonnes of paint."]


def test_support_score():
    assert support("The Louvre is the most visited museum [2].", _P[2]) == 1.0
    assert support("Bananas are yellow.", _P[2]) == 0.0

def test_first_use_order_and_short_sentences():
    answer = "Museums: the Louvre is the most visited museum [2]. The tower is 330 metres tall [1]. Yes."
    text, sources, unsupported = finalize_citations(answer, _P, _U)
    assert sources == ["https://b.example/louvre", "https://a.example/tower"]
    assert text.endswith("330 metres tall[2]. Yes.")
    assert unsupported == []                     # "Yes." is too short to be a claim

Lexical overlap is a cheap first filter. A small entailment model ("does this passage support this sentence?") on the remaining citations catches paraphrases and subtle mismatches; run it on a sample in production to measure citation precision, and on every answer for sensitive categories (health, finance).

Latency tricks

  • Stream status events immediately ("Searching…", then the sources list) so the page is alive before the first answer token.
  • Start the model as soon as the first good passages are reranked; do not wait for slow fetches.
  • Speculative search: run the likely search while the planner is still finishing, and keep it if the rewrite matches.
  • Prompt caching for the fixed instructions; keep passage order stable so the cache helps follow-ups.

Freshness and caching

The fresh tier crawls news sites and sitemaps every few minutes and indexes within minutes. Cache answers only for evergreen questions, keyed by the normalized query, with a time to live set by the freshness class; never cache answers to questions about the user's own context. Cache fetched pages by URL with a time to live per domain type.

Quality evaluation

A labeled set of questions graded for correctness, completeness and citation precision and recall, run on every change to planner, retrieval, prompt or model. In production: thumbs, copy and follow-up rates, reformulations (a sign the answer missed), and the share of sentences without a surviving citation.

Trade-offs

  • Snippets versus full pages. Snippets are instant and cheap; full pages give better answers but add latency and fetch cost. Fetch for complex questions, use snippets for simple ones.
  • Own index versus third-party search APIs. An own index gives control over freshness, ranking and cost at scale but is expensive to build; APIs are fast to start and costly per query. Most systems start with APIs and grow an index for the head of the traffic.
  • Small versus large answer models. Routing most queries to a small fine-tuned model cuts cost and latency; the router must be good at spotting hard questions, and both models need the same citation behavior.
  • Strict versus lenient citation checking. Strict removes more wrong citations but also some right ones and leaves more sentences uncited; tune the threshold against labeled data.

Interviewer expectations

Mid-level (L4)

  • A working pipeline: rewrite the query, search, fetch, pick passages, generate with citations, stream.
  • A sensible data model for conversations and sources, and follow-up handling.
  • Mentions timeouts on fetching.

Senior (L5)

  • A latency budget per stage that adds up to the target, with parallel fetching, deadlines and fallbacks.
  • Citation verification after generation: invalid numbers, unsupported citations, merging per URL, flagging uncited claims.
  • Load and cost numbers: queries per second, fetches per second with cache hit rates, tokens per answer.

Staff (L6+)

  • Freshness tiers, caching policy by question class, and the own-index versus API trade-off as the product scales.
  • Model routing between small and large models and how it is evaluated.
  • An evaluation program for correctness and citation precision, offline and online, that gates changes.

Sign in to comment, vote or remix.