Question: Design a RAG system over company docs
Companies: Anthropic, Databricks, Cohere, Perplexity
Difficulty: Medium
Problem statement
Employees should be able to ask questions about internal documents (Drive, Confluence, Slack, GitHub) and get grounded answers with citations.
Requirements
Functional requirements
- Employees ask questions in natural language and get answers grounded in internal docs, with citations.
- Respect document permissions: never show content a user can't open in the source system.
- Docs from Drive, Confluence, Slack and GitHub stay fresh within an hour.
Non-functional requirements
- Answer starts streaming within 2 s; 50k employees, 5 questions/day each.
- Measurable quality: a held-out set of questions with graded answers.
Back-of-the-envelope estimation
- 20M documents → ~200M chunks of ~300 tokens. 1024-dim float16 embeddings ≈ 2 KB each → ~400 GB of vectors, so an ANN index with sharding or quantization.
API design
POST /ask {question, conversation_id} → stream of {delta} then {citations: [{doc_id, url, snippet}]}.
Data model
chunks(id, doc_id, text, embedding, acl_principals, updated_at)documents(id, source, url, title, acl, version)- Vector index (HNSW or IVF-PQ) plus a keyword (BM25) index over the same chunks.
High-level design
flowchart LR
SRC[Connectors] --> ING[Ingest: parse, chunk, embed] --> IDX[(Vector + keyword index)]
Q[Question] --> RW[Query rewrite] --> RET[Hybrid retrieval + ACL filter] --> RR[Reranker] --> LLM[LLM with citations]
IDX --> RET
Deep dives
Chunking. Split on document structure (headings, paragraphs) with ~10% overlap; keep the title and heading path in each chunk's text so it stands alone.
Retrieval. Hybrid: BM25 catches exact names and error codes, embeddings catch paraphrases. Merge with reciprocal rank fusion, then a cross-encoder reranks the top 50 down to 8.
Permissions. Store ACL principals on each chunk and filter inside the index query (pre-filter), not after; sync ACL changes faster than content changes.
Grounding. Prompt the model to answer only from the provided chunks and cite chunk ids; post-check that every cited id was in the context. Say "I don't know" when retrieval scores are low.
Evaluation. Track retrieval recall@k on labeled questions and answer faithfulness (LLM-graded plus human spot checks) on every change to chunking, models or prompts.
Trade-offs
- Bigger chunks give more context per hit but dilute embeddings; smaller chunks need more of them per answer.
- Pre-filtering by ACL can hurt ANN recall when a user can see few docs; fall back to keyword search for those users.