What is LLM-as-a-judge?

LLM-as-a-judge means using one AI model to grade another AI model's answers — a fast, cheap substitute for hiring human reviewers to score thousands of responses.

It exists because grading open-ended text is hard. A math answer is simply right or wrong, but "summarize this contract" has no single correct output — someone has to judge whether the summary is accurate, complete, and clear. Humans are slow and expensive at that scale, so teams hand the rubric to a strong model instead.

In practice it works one of two ways:

• Direct scoring — the judge reads a question and an answer, then scores it against a rubric, say 1 to 10 for accuracy and helpfulness.
• Pairwise comparison — the judge sees two answers to the same question and picks the better one, which is how several benchmarks rank models.

The catch: judges have known biases. They tend to favor longer, more confident-sounding answers, prefer writing that resembles their own style, and in side-by-side tests can lean toward whichever answer appears first.

Good setups fight this by swapping answer order, using tight rubrics, and spot-checking a sample against human graders. Done carefully, it has become a standard tool in LLM evaluation — including for RAG systems, where a judge checks whether answers actually stick to the retrieved documents.