What is an LLM benchmark?

An LLM benchmark is a standardized test used to measure how well a large language model performs at a specific skill — every model answers the same fixed set of questions, so their scores can be compared directly.

Think of it as the SAT for AI. Instead of arguing about which model "feels smarter," a benchmark gives you a number you can put in a table and defend.

A few benchmarks get quoted constantly:

• MMLU — multiple-choice general knowledge across dozens of school and professional subjects.
• SWE-bench — whether a model can fix real bugs pulled from real GitHub projects.
• Math and reasoning suites — word problems and competition-style questions that test step-by-step thinking.

Here's the honest caveat: benchmarks age badly. Test questions leak into training data so models can partly memorize the answers, labs tune their models for the famous tests, and today's frontier models score so close together that a point or two rarely means anything you'd notice in daily use.

That's why a single score should never settle an argument. Serious comparisons pair benchmark results with leaderboards and hands-on evaluation on your own tasks — or you can compare current scores side by side in the model tracker.