1. Define the question set first
Write the questions your application must answer. Include questions whose answer is missing. This becomes the evaluation set and prevents you from choosing a database or embedding model before you know what “good retrieval” means.
2. Prepare documents with metadata
- Keep the source title, URL, owner, version, and effective date.
- Remove navigation, duplicate boilerplate, and obsolete versions.
- Split content at meaningful boundaries rather than arbitrary character counts.
- Keep enough context for a passage to make sense on its own.
3. Retrieve, filter, and ground
Retrieve more candidates than you plan to show, then filter by permissions, date, or document type. Tell the model to answer from the supplied passages, cite the supporting source, and say when the passages do not answer the question.
4. Evaluate before adding complexity
| Layer | Useful measurement |
|---|---|
| Retrieval | Recall@K or whether the supporting passage appears in the candidates |
| Grounding | Whether each material claim is supported by the context |
| Answer | Usefulness, completeness, and correct abstention |
| Operations | Latency, cost, document updates, and permission failures |
The first production safeguard
Log the retrieved source IDs and the final answer version. Without that trail, a bad answer becomes a debate about the model instead of a debuggable failure in ingestion, retrieval, prompting, or review.