Nearly every AI model announcement now comes with a slide of benchmark scores — a model "scoring 81% on SWE-bench" or "topping the coding leaderboard." These numbers get repeated constantly, but rarely explained. Here's what's actually being measured, and how much stock to put in it.

The Essentials

  • Every AI model launch comes with a chart of benchmark scores.
  • Here's a plain-English guide to what those numbers actually test, why they climb so fast, and how much weight they really deserve.
  • Nearly every AI model announcement now comes with a slide of benchmark scores — a model "scoring 81% on SWE-bench" or "topping the coding leaderboard." These numbers get repeated constantly, but rarely explained.

What a coding benchmark actually tests

Take one of the most-cited benchmarks in AI coding right now, SWE-bench. It works by taking real, historical bug reports from open-source software projects on GitHub — issues that a human developer already fixed at some point. The model is given the codebase and the issue description, and has to produce a code patch. That patch is then run against the project's own automated tests. If the tests pass, the model gets credit; if not, it doesn't. A score of 80% means the model successfully fixed roughly 4 out of 5 of those real-world issues on its own.

That's meaningfully different from a benchmark that just asks a model to write a short function from a description — those tests measure syntax and isolated logic, not the harder skill of navigating a large, unfamiliar codebase and making a fix that doesn't break anything else.

A high benchmark score says a model is good at benchmark-style tasks. It doesn't automatically say it's good at your task.

Why scores climb so fast — and why that's not the whole story

Benchmark scores across the industry have risen sharply over the past couple of years, and competition between labs is part of why: a new model that beats the leaderboard is a marketing moment, so labs have a strong incentive to optimize for the specific tasks a popular benchmark contains. That creates a real risk that models get tuned to do well on the test itself rather than on the broader skill the test was meant to represent — a well-known problem sometimes called benchmark overfitting.

There's also a more basic issue: popular benchmarks are public, and their source material — old GitHub issues, common textbook problems — may well have been part of a model's training data. When that happens, a high score can partly reflect memorization rather than fresh problem-solving, which is why researchers keep releasing new, harder versions of these tests.

How to actually use these numbers

Benchmark scores are most useful as a rough, relative signal — comparing models against each other on the same test, at the same point in time — not as a promise about how a tool will perform on your specific problem. A model that tops a coding leaderboard may still struggle with a task the benchmark never covered, like understanding your team's internal conventions or working across an unusually large codebase. Treat benchmark scores the way you'd treat a single standardized test score for a job candidate: informative, but only one input among several.