October 12, 2026
FrontierCode 1.1: the best model scores 52% on a test that checks whether a maintainer would accept the change
Claude Opus 5.5 (max with fallback) leads with 52%. In 58% of attempts it passes all blocking checks, and failing any one of them zeroes the attempt. Nine of 100 tasks were not solved by any model.
Artificial Analysis
@artificialanlys
Launching the FrontierCode 1.1 leaderboard: a closed evaluation from Cognition. It tests whether models can solve coding tasks in a way that maintainers of real open source repositories would accept the change. @cognition announced FrontierCode in June 2026, and version 1.1 came out in July. Cognition built the dataset together with open source maintainers, and each task took them more than 40 hours. The agent must make a change to a real open source repository. Then both unit tests and quality criteria are checked: what a maintainer would require before accepting the change. We run FrontierCode 1.1 Main, the core set of 100 tasks out of 150, through our open reference harness Stirrup. This way models are compared on equal terms. All tasks are held out so they cannot be seen during model training. Each attempt is graded on blocking and non-blocking criteria. Failing at least one blocking criterion gives a score of zero. If all pass, the attempt gets from 70% to 100%. What sets it apart from other code evaluations is a broader view of success, based on real use. Models must pass unit tests, but they are also judged on behavior: excessive changes and large diffs, edits to files outside the needed scope, and useless tests. We think this matters now: models already write code very well, but their changes often need serious rework before a maintainer accepts them. ➤ Leading models still fail more than 40% of attempts. Claude Opus 5.5 (max with fallback) leads with 52%, and it passes blocking checks in 58% of its attempts. Nine of the 100 tasks were not solved by any of the models we evaluated at launch. ➤ The top of the table is tight: nine models from six labs score between 43% and 52%. The leader among open-weight models is MiMo-V2.6-Pro at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44%. ➤ GPT-6.1 Sol (max) offers the best ratio of price to result: at $0.81 per task it scores 49%, the same as Claude Fable 5.1 (max with fallback). That is the level of Anthropic's leading models at less than a tenth of the cost per task. Thanks to @silasalberti, @ybenpan, @cl571128 and the rest of the Cognition team for FrontierCode and for the collaboration that brought it to Artificial Analysis. We will add FrontierCode to our coding agent leaderboard soon, and we are already working on it.
· 81.9K views
Usual code benchmarks focus on unit tests. FrontierCode also looks at behavior: unnecessary edits, large diffs, changes outside the needed scope, and useless tests.
Cost per task. GPT-6.1 Sol (max) scores 49% at $0.81 per task, the same as Claude Fable 5.1 (max with fallback). Anthropic's leading models cost more than ten times as much per task as Sol.
Open weights. MiMo-V2.6-Pro leads among open-weight models with 45%. DeepSeek V4.1 Flash (max) and Kimi K3 (max) score 44% each.
Rerunning the benchmark. The tasks are held out, so you cannot run your own evaluation on them. The Stirrup harness that Artificial Analysis uses to run models is open and can be installed with pip install stirrup.
FrontierCode will join the Artificial Analysis coding agent leaderboard. The tweet says it will happen "very soon," with no date.