Start with the missed question
Take 20 questions the system answered poorly and label the failure. Was the answer in the corpus? Was the right document retrieved? Was the passage too broad? Did a date or permission filter remove it? Without this diagnosis, retrieval tuning becomes guesswork.
The repair order
- Clean duplicate, stale, and contradictory documents.
- Split content around headings and meaning, not only token count.
- Keep document titles, dates, and section names in the searchable context.
- Rewrite vague user questions into a search query when needed.
- Use metadata filters for tenant, product, date, and document status.
- Add reranking only after the candidates are relevant enough to rerank.
Measure retrieval directly
For each test question, record whether the supporting passage appears in the top K results. Then measure whether the final answer uses it correctly. A fluent answer can hide a retrieval failure, so keep these metrics separate.
A common false fix
Increasing the number of retrieved chunks can make a demo look better while making production answers less focused. More context increases cost and can bury the one passage that matters. Improve candidate quality before increasing context volume.