Design a RAG system over company docs
Remix this answer and make it yours, or start blank. Either way you work beside the reference, one chapter at a time, and it stays private until you publish it.
Problem statement
Employees should be able to ask questions about internal documents (Drive, Confluence, Slack, GitHub) and get grounded answers with citations.
Requirements
Functional requirements
- Employees ask questions in natural language and get answers grounded in internal docs, with citations.
- Respect document permissions: never show content a user can't open in the source system.
- Docs from Drive, Confluence, Slack and GitHub stay fresh within an hour.
Non-functional requirements
- Answer starts streaming within 2 s; 50k employees, 5 questions/day each.
- Measurable quality: a held-out set of questions with graded answers.
Back-of-the-envelope estimation
- 20M documents → ~200M chunks of ~300 tokens. 1024-dim float16 embeddings ≈ 2 KB each → ~400 GB of vectors, so an ANN index with sharding or quantization.
API design
POST /ask {question, conversation_id} → stream of {delta} then {citations: [{doc_id, url, snippet}]}.
Data model
chunks(id, doc_id, text, embedding, acl_principals, updated_at)documents(id, source, url, title, acl, version)- Vector index (HNSW or IVF-PQ) plus a keyword (BM25) index over the same chunks.
High-level design
flowchart LR
SRC[Connectors] --> ING[Ingest: parse, chunk, embed] --> IDX[(Vector + keyword index)]
Q[Question] --> RW[Query rewrite] --> RET[Hybrid retrieval + ACL filter] --> RR[Reranker] --> LLM[LLM with citations]
IDX --> RET
Deep dives
Chunking. Split on document structure (headings, paragraphs) with ~10% overlap; keep the title and heading path in each chunk's text so it stands alone.
Retrieval. Hybrid: BM25 catches exact names and error codes, embeddings catch paraphrases. Merge with reciprocal rank fusion, then a cross-encoder reranks the top 50 down to 8.
Permissions. Store ACL principals on each chunk and filter inside the index query (pre-filter), not after; sync ACL changes faster than content changes.
Grounding. Prompt the model to answer only from the provided chunks and cite chunk ids; post-check that every cited id was in the context. Say "I don't know" when retrieval scores are low.
Evaluation. Track retrieval recall@k on labeled questions and answer faithfulness (LLM-graded plus human spot checks) on every change to chunking, models or prompts.
Trade-offs
- Bigger chunks give more context per hit but dilute embeddings; smaller chunks need more of them per answer.
- Pre-filtering by ACL can hurt ANN recall when a user can see few docs; fall back to keyword search for those users.