Which neural network is best for coding: a guide to benchmarks for AI programmers

A coding benchmark is an exam where the answer is checked by tests, not a person. The model gets a task, writes code, and the code is run. It passes every check, the task is solved. Even one failed test means it is not solved, with no “almost right.”

That makes coding benchmarks more objective than any other exam. A history essay can be graded differently, but you cannot argue with a failing test.

All follow the same general mechanics: a set of tasks, a harness that feeds them to the model, and a final score. A harness is the execution wrapper: code that determines which tools and how many attempts the model gets. We covered how any benchmark works and why a number without its passport means nothing in our in-depth benchmark guide. Here, we look only at code.

Coding benchmarks have their own case history. First, models memorize the tasks. Then it turns out the harness shapes the score. And in summer 2026, an even subtler catch surfaced: sometimes a different model answers for the model. We will cover all three in order, through real cases.

OpenAI released the first mainstream coding benchmark in July 2021. HumanEval, 164 Python tasks: here is a function description, write the body, pass the tests. The Codex of the time solved 28,8% (OpenAI article, 2021).

It seemed like a measure of progress for years. It lasted four. By 2025, top models reached 96%+, and the difference between “97” and “98” across 164 tasks is fewer than two tasks — random noise. The benchmark stopped distinguishing the strong models.

The 2021 tasks have also been online for years, meaning they are in model training data. On HumanEval, you can no longer distinguish “solved it” from “remembered it.” If a review praises a neural network with a HumanEval number, the review is about three years out of date.

The moral of the first story: a small set of function tasks is not the profession. A programmer does not write functions from descriptions in a vacuum; they dig through someone else's repository. The industry understood this and built a benchmark from real work.

In October 2023, a Princeton team assembled SWE-bench from 2 294 real GitHub tasks: live bug reports and feature requests from 12 popular Python projects. Models get the entire repository and the task text. They must write a patch, which is run against hidden tests.

In 2024–2026, researchers dissected SWE-bench from three angles, and every cut gave less reason to trust the table.

The answers were in the tasks themselves. SWE-bench tasks are old issues from public repositories, and many discussions directly explain how to fix them. When researchers filtered out such tasks and weak tests, agent performance on Verified fell from 51,7% to 25,9% (SWE-Bench+, 2024). Half the success relied on hints and leaky checks.

The model memorizes the tasks. The paper “The SWE-Bench Illusion” (June 2025) showed that top models name the file containing the bug in 76% of cases from the task text alone, without looking at the repository at all. On projects outside the benchmark, the same ability falls to 53%. That is memory, not engineering instinct. The tasks were simply in the training data.

The corpus is narrow. Around 40% of tasks come from one project, Django, and five projects account for more than 80% of tasks. And it is all Python. A top performer on this benchmark knows old Django bugs from someone else's codebase, not the programming profession.

Then something rare happened: the industry agreed. In early 2026, OpenAI publicly stopped reporting Verified results and switched to next-generation benchmarks. That is the next story.

There is one answer to memorized tasks: tasks the model could not have seen. Three approaches.

Closed tasks. SWE-bench Pro (Scale AI, September 2025) assembled 1 865 tasks from 41 repositories in four languages, and keeps part of the corpus private: those repositories could not have entered training. The tasks are substantial, averaging a 107-line change across four files. The result was sobering.

Fresh tasks by date. LiveCodeBench only counts tasks published after the end of a model's training. There is physically nothing to copy from. Its sibling, LiveCodeBench Pro, took 584 Olympiad-level tasks: the best models without tools solve 53% of medium and 0% of hard tasks (June 2025). On the Olympiad scale, that is master level, far from grandmaster.

Real money instead of points. SWE-Lancer (OpenAI, February 2025) took over 1,400 real freelance jobs from Upwork, worth a million dollars in total payouts, from a $50 bug fix to a $32,000 feature. The study's best model "earned" $403,000 out of a million. A good apprentice, but not yet one to trust with most jobs.

There is also a benchmark for end-to-end autonomy. Terminal-Bench tests how a model handles a task itself in the terminal from start to finish, including environment setup and debugging. On its hard tier, the best score around 66% (Artificial Analysis, May 2026).

Let's put everything into one table. The names are links to the platforms themselves, and GitHub stars show how well known a benchmark is among developers.

The map looks like an answer. Now for the second catch: the score in the table belongs not to the model, but to the "model + harness" pair.

The best proof is the official Terminal-Bench leaderboard, where the same model is listed in several harnesses at once.

The same from another angle. The Claude Opus line has three numbers on the same SWE-bench Verified: 76.8% on the official leaderboard in a modest bash harness, 80.9% in Anthropic's release claim with its own harness (Opus 4.5, November 2025), and 88.6% from independent evaluator Vals AI (Opus 4.8, August 2026). Nobody is lying. They are simply three different test conditions and two model versions.

The knob is even effort. One gpt-5 on Aider Polyglot: 88.0% at high reasoning effort and 81.3% at low - seven points from one toggle.

This leads to a reading rule. A table row without the harness name says nothing, and changing the harness legitimately moves the score and changes the winner: when aggregator Artificial Analysis replaced one component of its index, the leader changed without a single new model release.

The third story happened in summer 2026 and raised a question nobody had considered before: who was actually answering the tasks.

Anthropic released Claude Fable 5 with an unusual setup: a classifier filter sits in front of the model, and it sends requests from sensitive areas to another model, Opus 4.8. In the API, this refusal is clearly visible in a separate response field. But in apps and in Claude Code, the substitution happens quietly: you ask one model, another answers.

For evaluators, this became a trap. Artificial Analysis found that Opus 4.8 answered instead of Fable 5 on roughly 8% of its index tasks, and published a mixed score, honestly labeling the configuration "Opus 4.8 Fallback." Vals AI showed the range of interpretations on the GPQA Diamond science benchmark: 93.18% if answers from the substitute model count, and 55.56% if every refusal counts as a failure. One model, one table, thirty-eight points of difference from a single accounting policy.

Two days after release, Anthropic made the quietest of the filters visible, and evaluators split scores into "pure" and "practical": the model's pure capability and behavior with substitutions. Both numbers are honest, but they are different numbers.

This gave a score's passport a fourth question: whose responses are inside the score. A good evaluator answers it directly in the configuration name.

The most uncomfortable measurement came from the METR lab in summer 2025, and it is about people. 16 experienced developers solved 246 tasks in their own projects, with a random portion of tasks done with AI assistants. Before starting, they expected a 24% speedup. After finishing, they rated their speedup at 20%. The stopwatch showed the opposite: with AI, they worked 19% slower.

This is early 2025, specific tools, and very experienced people working in code they knew very well, and METR is already redesigning the experiment (February 2026). But the conclusion for reading tables is timeless. A benchmark score and usefulness in your project are different quantities, and the latter does not automatically follow from the former.

What code benchmarks barely measure at all: reviewing someone else's code, debugging from a poor bug report, working in a million-line legacy codebase, readability and security of what is written. In other words, a large part of a programmer's real work.

Now let's assemble everything into a short selection method.

1. Identify your task class; each has its own benchmark.

Benchmark debunkings are not a verdict on models. Honest measurements of progress are more impressive than marketing.

METR measures not scores, but task length: what work, measured in expert human hours, a model can handle with 50% reliability. By January 2026, the best model's horizon had reached over five hours, and it doubles every 4-7 months (METR, Time Horizon 1.1). A year earlier, it was measured in tens of minutes.

At ICPC 2025, the world final of team programming, OpenAI models solved 12 out of 12 tasks. No human team solved that many. Gemini solved 10 out of 12, including a task no human team could solve (September 2025).

There is no contradiction with "0% hard tasks" here. At the Olympiad, models were given time, attempts, and a compute budget; in the benchmark, they got one attempt without tools. That is the core skill for reading code numbers: a score exists only together with its measurement conditions.

1. A code benchmark is an exam checked by tests. It is more honest than an essay, but it has three catches: learned tasks, the harness, and substitution of the test taker.

2. A number needs a passport with four questions: who measured it, with which harness, when the tasks were collected, and whose responses are inside the score.

3. A high score on SWE-bench Verified does not make a model a programmer: on fresh closed tasks, the same models solve one out of six.

4. Choose from the map based on your task class, take the top 3, and run them on a couple of your own tasks: they definitely were not in the training data.

5. Progress is real; the task horizon doubles every 4-7 months. It is not models that lie, but numbers without measurement conditions.