Why does my RAG return wrong or irrelevant answers?

Almost always, the problem is retrieval, not the model — the right chunk never made it into the prompt, so the LLM answered from the wrong context or from memory. Bad chunking, a mismatched embedding model, no reranking, and distracting chunks are the usual culprits. A smaller share of failures happen at generation, where the model ignores good context or fills gaps with confident guesses.

How do I tell retrieval from generation failures?

Run one quick test: look at the chunks your system actually retrieved for the bad answer. If the correct information isn't in them, it's a retrieval problem — no model can answer from context it never received. If the right chunk is there and the model still got it wrong, it's a generation problem.

This single check tells you which half of the pipeline to fix and saves hours of guessing.

Retrieval checklist

• Bad chunking — the answer is split across a chunk boundary or buried in a chunk too big to match well. Revisit chunk size and overlap.
• Wrong embedding model — it misses exact terms (IDs, acronyms, code) or doesn't fit your domain. Add keyword or hybrid search, or test a better model.
• No reranking — the right chunk is retrieved but sits at rank 9, so the LLM never leans on it. Add a reranker.
• Too many chunks — stuffing 20 chunks in the prompt buries the good one, and irrelevant distractor chunks actively pull the model off track. Retrieve wide, then keep only the top few.
• Missing content — the answer simply isn't in your index. No retrieval trick fixes a gap in the source data.

Generation checklist

• The model answers from its training data instead of the context — tighten the prompt to say answer only from the provided context.
• It hedges or makes things up when context is thin — instruct it to say it doesn't know when the context lacks the answer.
• Conflicting chunks confuse it — dedupe your source data and rerank so the most authoritative chunk ranks first.

Work top to bottom and fix retrieval first, because even a great model can't rescue bad context.