What are RAG evaluation metrics?

RAG evaluation comes down to four questions: did it find the right documents, did the answer actually stick to what those documents say, did the answer address what the user asked, and was it fast and cheap enough to ship. Each one needs its own check, because a RAG system can fail at any single step while looking fine everywhere else.

What are the core RAG evaluation metrics?

The four checks map to the four ways a RAG system quietly breaks:

• Retrieval quality — did you pull back the documents that actually answer the question? Measured with precision (of what you retrieved, how much was relevant) and recall (of what was relevant, how much did you actually retrieve).
• Faithfulness (groundedness) — does the generated answer stick to what's in the retrieved documents, or did the model quietly add something the sources never said? This is the metric that catches confident-sounding hallucinations.
• Answer relevance — even a perfectly grounded answer is useless if it doesn't address what the user actually asked. This checks the answer against the question, not against the source documents.
• Latency and cost — a system that's accurate but takes ten seconds and burns real money per query isn't shippable. Speed and price are part of "good," not an afterthought.

Why measure retrieval and generation separately?

Because they fail independently. If retrieval hands the model the wrong documents, even a perfect model can only give a wrong answer — that's a retrieval problem. If retrieval is spot-on but the model still invents facts, that's a generation problem.

Score them together and a good half hides a bad one, so you never learn which part to fix. Splitting the metrics tells you exactly where to spend your time.

Can you automate RAG evaluation?

Partly. You can check faithfulness and relevance by hand on a sample of queries, but that doesn't scale past a handful of test cases. Frameworks like RAGAS exist to automate these checks, typically by using another LLM as a judge to score faithfulness and relevance against the retrieved context.

Treat those automated scores as a fast way to catch regressions, not as gospel — spot-check them against human judgment every so often so you know the judge itself is still trustworthy.