Which AI is better: a look at the exams that measure AI

A benchmark is an exam for an AI model. Its creators assemble a set of tasks with verifiable answers, run models through it, and publish a score table. No magic. The same Unified State Exams, except ChatGPT, Claude, and DeepSeek take them instead of schoolchildren.

The word is overloaded. In search, “benchmark” most often means a hardware test like AnTuTu for a smartphone. Here, we mean exams for AI, which work differently and have their own problems.

A benchmark is not the same as a ranking. A benchmark gives a score for one exam, while a ranking brings together many exams in a “who is better” table. Behind every “AI model ranking 2026” are specific benchmarks, and a ranking’s quality depends on their quality.

The Unified State Exam analogy is not literal. A human exam has a proctor and the test taker’s ID, while a machine exam has only the organizers’ honesty. You will see where that leads.

Every benchmark consists of three parts.

Benchmarks have short lives. Models surpassed GLUE, the main 2018 language-understanding exam, in about a year. MMLU lasted four years: GPT-3 scored 43.9% in 2020, while by 2024 the leaders reached 88% and stopped differing from one another.

Stanford AI Index records that fresh benchmarks now saturate within months. A saturated exam is useless: a test for fifth graders cannot distinguish between two professors, since both will get an A.

The industry’s first answer is ever harsher datasets such as FrontierMath. More than sixty mathematicians, including Terence Tao, contributed its problems.

The second answer is composite indices. Epoch Capabilities Index stitches more than fifty benchmarks into one scale, and it does not break when yet another exam is “solved.”

There is no universally best model, only the best one for the task. A coding champion does not have to be a video or image champion. The tables confirm it: leaders across different benchmark classes do not match.

A ten-minute algorithm.

1. Define your task: code, text, images, or math.

2. Open the relevant benchmark from the map above and choose the top 3, not the top 1. The difference within the top three is usually smaller than the measurement error.

3. Run all three models on a couple of your real tasks. Your own mini-exam will tell you more than someone else’s table.

If you do not want to assemble tables by hand, look at composite indices: the provider tracks harnesses and dates there.

For code, we do it ourselves: RuBench runs coding agents on Russian-language tasks using an open methodology.

Benchmarks have long become marketing: a high ranking sells subscriptions.

Model swapping in the arena. In April 2025, Meta* (recognized in Russia as an extremist organization; its activities are banned) submitted a special version of Llama 4 Maverick to LMArena, trained to appeal to people in duels. The custom version took second place, while the public one ranked around thirty-second. After the scandal, the arena tightened its rules.

The best-attempt game. The study “The Leaderboard Illusion” (April 2025) showed that major labs privately ran dozens of model variants in the arena and published only the best one. Meta* ran up to 27 variants before releasing Llama 4.

Training-data leakage. Models learn from internet texts, including exam questions themselves. Testing on a “clean mirror” lowered GSM8K accuracy by up to 23 points (NAACL 2024). The model was not solving the problems, it remembered them: the questions leaked online long before the exam.

Defects in the exams themselves. In MMLU, 6.5% of questions contain errors (the study “Are We Done with MMLU?”, 2024). SWE-bench was repaired through manual selection of five hundred “validated” tasks, but a subsequent review found serious defects in 59% of failed runs. After that, OpenAI publicly wrote off this benchmark as a frontier metric in 2026.

Pretty release charts. At the GPT-5 presentation in August 2025, the chart bars contradicted their own numbers, and Sam Altman admitted a “mega chart screwup.” An announcement slide is not a source. An independent run is the source.

In 2025, the METR lab tested this with real people. 16 experienced developers solved 246 real tasks in their own projects, some with AI assistants. With AI, they were 19% slower, although they themselves estimated their speedup at plus 20%.

The conclusion is not an indictment of AI models, but of blind faith in scores. An exam measures a task in a vacuum. In real work, project context, skill at defining tasks, and the cost of reviewing someone else’s code matter.

Every table has an author, and every author has interests.

Epoch AI, a nonprofit lab funded by grants, maintains FrontierMath. In January 2025, it admitted that OpenAI had secretly paid for the benchmark’s creation and had access to the tasks.

LMArena grew from an academic Berkeley project into a company valued at 1.7 billion dollars. Its growth coincided with the main criticism: a study of the arena’s bias toward major labs was published back in April 2025.

Scale AI, which ran the SEAL leaderboards, sold 49% of the company to Meta* in 2025. Competitors stopped working with it.

Crowd also measures. Millions of arena votes honestly measure preferences; the only question is what you want to buy: an appealing answer or a solved task.

The labs themselves publish scores for their models in their own harnesses, and these figures do not undergo independent validation by default.

The reading rule is simple. For every number, look for the trio: who measured it, in which harness, on what date. If even one answer is missing, you are looking at advertising, not fact.

English exams barely test Russian, so it has its own. MERA is an open benchmark by a consortium of Sber, Yandex, Skoltech, and the Russian Academy of Sciences, with 21 tasks and closed tests. llmarena.ru runs blind duels in Russian. GigaChat, YandexGPT, and T-pro models compete there.

Our contribution in this category is RuBench, an independent benchmark of coding agents on Russian-language tasks with an open methodology.

1. A benchmark is an exam for an AI model. A rating that does not name the benchmarks behind it is empty.

2. A number exists only with the trio: who measured it, which harness, which date. Without the trio, it is advertising.

3. One overall score solves nothing. Choose by your task class and take the top 3, not the top 1.

4. The arena shows the crowd’s taste, an exam measures knowledge, and a composite rolls everything into one score. These are three different tools.

5. The best exam is your own. Ten minutes with three models on your task will tell you more than any table.