Which neural network is best for coding: a guide to benchmarks for AI programmers

A coding benchmark is an exam where the answer is checked by tests, not a person. The model gets a task, writes code, and the code is run. It passes every check, the task is solved. Even one failed test means it is not solved, with no “almost right.”

That makes coding benchmarks more objective than any other exam. A history essay can be graded differently, but you cannot argue with a failing test.

All follow the same general mechanics: a set of tasks, a harness that feeds them to the model, and a final score. A harness is the execution wrapper: code that determines which tools and how many attempts the model gets. We covered how any benchmark works and why a number without its passport means nothing in our in-depth benchmark guide. Here, we look only at code.

Coding benchmarks have their own case history. First, models memorize the tasks. Then it turns out the harness shapes the score. And in summer 2026, an even subtler catch surfaced: sometimes a different model answers for the model. We will cover all three in order, through real cases.

OpenAI released the first mainstream coding benchmark in July 2021. HumanEval, 164 Python tasks: here is a function description, write the body, pass the tests. The Codex of the time solved 28.8% (OpenAI article, 2021).

It seemed like a measure of progress for years. It lasted four. By 2025, top models reached 96%+, and the difference between “97” and “98” across 164 tasks is fewer than two tasks — random noise. The benchmark stopped distinguishing the strong models.

The 2021 tasks have also been online for years, meaning they are in model training data. On HumanEval, you can no longer distinguish “solved it” from “remembered it.” If a review praises a neural network with a HumanEval number, the review is about three years out of date.

The moral of the first story: a small set of function tasks is not the profession. A programmer does not write functions from descriptions in a vacuum; they dig through someone else's repository. The industry understood this and built a benchmark from real work.

In October 2023, a Princeton team assembled SWE-bench from 2,294 real GitHub tasks: live bug reports and feature requests from 12 popular Python projects. Models get the entire repository and the task text. They must write a patch, which is run against hidden tests.

In 2024–2026, researchers dissected SWE-bench from three angles, and every cut gave less reason to trust the table.

The answers were in the tasks themselves. SWE-bench tasks are old issues from public repositories, and many discussions directly explain how to fix them. When researchers filtered out such tasks and weak tests, agent performance on Verified fell from 51.7% to 25.9% (SWE-Bench+, 2024). Half the success relied on hints and leaky checks.

The model memorizes the tasks. The paper “The SWE-Bench Illusion” (June 2025) showed that top models name the file containing the bug in 76% of cases from the task text alone, without looking at the repository at all. On projects outside the benchmark, the same ability falls to 53%. That is memory, not engineering instinct. The tasks were simply in the training data.

The corpus is narrow. Around 40% of tasks come from one project, Django, and five projects account for more than 80% of tasks. And it is all Python. A top performer on this benchmark knows old Django bugs from someone else's codebase, not the programming profession.

Then something rare happened: the industry agreed. In early 2026 OpenAI publicly stopped reporting Verified scores and moved to next-generation benchmarks.

By fall the rest followed. The independent evaluator Vals AI moved Verified to its archive and no longer runs new models on it: seven models had already scored 95% or higher there (last updated Sep 1, 2026). In the Claude Opus 5.5 release of Sep 22, the Verified row is already gone. The next story covers the new generation of benchmarks.

There is one answer to memorized tasks: tasks the model could not have seen. Three approaches.

Closed tasks. SWE-bench Pro (Scale AI, September 2025) assembled 1,865 tasks from 41 repositories in four languages, and keeps part of the corpus private: those repositories could not have entered training. The tasks are substantial, averaging a 107-line change across four files. The result was sobering.

The gap survived a change of generations. Claude Opus 5, Fable 5 and GPT-5.6 Sol score 95–97% on Verified (Vals AI, September 2026). On the 113 tasks of the DeepSWE benchmark, written from scratch, the same models solve 70–74% (Datacurve, Sep 22, 2026). The harness in both measurements is equally modest: the model has only a terminal.

Fresh tasks by date. LiveCodeBench only counts tasks published after the end of a model's training. There is physically nothing to copy from. Its sibling, LiveCodeBench Pro, took 584 Olympiad-level tasks: the best models without tools solve 53% of medium and 0% of hard tasks (June 2025). On the Olympiad scale, that is master level, far from grandmaster.

Real money instead of points. SWE-Lancer (OpenAI, February 2025) took over 1,400 real freelance jobs from Upwork, worth a million dollars in total payouts, from a $50 bug fix to a $32,000 feature. The study's best model "earned" $403,000 out of a million. A good apprentice, but not yet one to trust with most jobs.

There is also a benchmark for autonomy as a whole. Terminal-Bench checks how a model runs a task in the terminal on its own from start to finish, setting up the environment and debugging. Since Aug 27, 2026 version 4.0 has been in force, with 66 tasks harder than the previous ones. The best pair on the official leaderboard, Codex with GPT-6 Astra, solves 58% (tbench.ai, Sep 26, 2026).

Let's put everything into one table. The names are links to the platforms themselves, and GitHub stars show how well known a benchmark is among developers.

The map looks like an answer. Now for the second catch: the score in the table belongs not to the model, but to the "model + harness" pair.

The design of DeepSeek Harness shows what surrounds a model: folder access, commands, permissions and plugins. It describes the execution environment, not one more score for the DeepSeek model.

The best proof is the official Terminal-Bench leaderboard, where the same model is listed in several harnesses at once.

The same thing from another angle. The Claude Opus line has three numbers on one and the same SWE-bench Verified: 76.8% on the official leaderboard in a modest bash harness, 80.9% in Anthropic's release claim with its own harness (Opus 4.5, November 2025) and 97% at independent evaluator Vals AI (Opus 5, September 2026). Nobody is lying. These are simply three different test conditions and two versions of the model.

A recent example is Opus 5.5 on Terminal-Bench 4.0. In its release of Sep 22 Anthropic claimed 66.4% in its own run, while Artificial Analysis measured 63.1% in the Claude Code harness (Sep 25, 2026). One model, the same tasks, different conditions.

The knob is even effort. One gpt-5 on Aider Polyglot: 88.0% at high reasoning effort and 81.3% at low - seven points from one toggle.

This leads to a reading rule. A table row without the harness name says nothing, and changing the harness legitimately moves the score and changes the winner: when aggregator Artificial Analysis replaced one component of its index, the leader changed without a single new model release.

The third story happened in summer 2026 and raised a question nobody had considered before: who was actually answering the tasks.

In June 2026 Anthropic shipped Claude Fable 5 with an unusual design: a classifier filter sits in front of the model, and it hands requests from sensitive areas to a different model, Opus 4.8. In the API such a handoff is shown explicitly, in a separate response field. In the apps and in Claude Code the substitution happens quietly: you asked one model, another one answered.

For evaluators, this became a trap. Artificial Analysis found that Opus 4.8 answered instead of Fable 5 on roughly 8% of its index tasks, and published a mixed score, honestly labeling the configuration "Opus 4.8 Fallback." Vals AI showed the range of interpretations on the GPQA Diamond science benchmark: 93.18% if answers from the substitute model count, and 55.56% if every refusal counts as a failure. One model, one table, thirty-eight points of difference from a single accounting policy.

Two days after release, Anthropic made the quietest of the filters visible, and evaluators split scores into "pure" and "practical": the model's pure capability and behavior with substitutions. Both numbers are honest, but they are different numbers.

Since then the substitution has become standard for the senior Claude models. Anthropic measured Opus 5.5, released on Sep 22, with the filters on. Cybersecurity tasks were finished for it by Opus 4.8, and biology tasks by Opus 5.

Artificial Analysis now counts substitutions in a separate column. In its agent index of Sep 25, the leader, Claude Code with Opus 5.5, triggered the filter on 80 attempts out of 909, and another model finished 78 of them.

This gave a score's passport a fourth question: whose responses are inside the score. A good evaluator answers it directly in the configuration name.

The most awkward measurement came from the METR lab in summer 2025, and it is about people. 16 experienced developers worked through 246 tasks in their own projects, with a random subset of tasks done with AI assistants. Before starting, they expected a 24% speedup. After finishing, they estimated their speedup at 20%. The stopwatch showed the opposite: with AI they worked 19% slower.

This is early 2025, specific tools and very experienced people working in code they know very well. A repeat measurement on late-2025 agents did show a speedup, but a modest one, 4–20% (METR, May 2026). METR itself considers the figure understated: some of the developers refused to take part because they do not want to work without AI.

The conclusion for reading tables does not change because of this. A benchmark score and usefulness in your project are different quantities, and the second does not follow from the first automatically.

What coding benchmarks barely measure at all: reviewing others' code, debugging from a messy bug report, working in a million-line legacy codebase, readability and security of the code written. In other words, a big chunk of a programmer's real work.

Now let's assemble everything into a short selection method.

1. Identify your task class; each has its own benchmark.

Benchmark debunkings are not a verdict on models. Honest measurements of progress are more impressive than marketing.

METR measures not scores but task length: how much work, counted in hours of a professional human, a model carries through with 50% reliability. Claude Opus 4.6 from February 2026 has a horizon of about twelve hours; Claude 3.7 Sonnet from February 2025 had about an hour. Since 2023 the horizon has been doubling roughly every four months (METR, data as of May 8, 2026).

At ICPC 2025, the world final of team programming, OpenAI models solved 12 out of 12 tasks. No human team solved that many. Gemini solved 10 out of 12, including a task no human team could solve (September 2025).

There is no contradiction with "0% hard tasks" here. At the Olympiad, models were given time, attempts, and a compute budget; in the benchmark, they got one attempt without tools. That is the core skill for reading code numbers: a score exists only together with its measurement conditions.

1. A code benchmark is an exam checked by tests. It is more honest than an essay, but it has three catches: learned tasks, the harness, and substitution of the test taker.

2. A number needs a passport with four questions: who measured it, with which harness, when the tasks were collected, and whose responses are inside the score.

3. A high score on SWE-bench Verified does not make a model a programmer: models that score 95–97% on it lose more than twenty points on tasks written from scratch.

4. Choose from the map based on your task class, take the top 3, and run them on a couple of your own tasks: they definitely were not in the training data.

5. The progress is real, the task horizon doubles roughly every four months. It is not the models that lie, it is numbers without the conditions of the measurement.