Start with one question class
Do not begin with “chat with all company data.” Choose a job such as “answer questions about the current refund policy” or “find the setup steps in our product docs.” A narrow job gives you a test set and a clear definition of wrong.
The basic pipeline
- Collect approved documents and keep their titles, URLs, owners, and dates.
- Split documents into passages that preserve enough surrounding context.
- Create embeddings and store them with document metadata.
- Retrieve the most relevant passages for a user question.
- Pass those passages to the model with an instruction to use only the supplied evidence.
- Return the answer with citations and a fallback when the evidence is insufficient.
A test question
For a support policy, test both “Can I pause my annual plan?” and “Can I pause a plan for a customer type the policy never mentions?” The first checks useful retrieval. The second checks whether the system knows when to stop instead of filling the gap with a plausible answer.
What to measure
| Layer | Question |
|---|---|
| Retrieval | Did the right passage appear in the top results? |
| Grounding | Does the answer stay inside the retrieved evidence? |
| Citations | Can a reader verify the exact claim? |
| Abstention | Does it refuse or ask for clarification when evidence is missing? |
| Operations | Can documents be replaced, dated, and removed safely? |
Do not add complexity too early
A larger vector database, agent loop, or reranker will not fix a source set with duplicated, stale, or badly chunked documents. Make the small pipeline reliable first. Complexity is justified only by a measured failure in the simpler version.