September 23, 2026
Running an AI agent patch is not enough: SlopCodeBench solved no task end-to-end
In SlopCodeBench's March 25 write-up, the best result was 14.8%, and no task was completed by an AI agent. The benchmark tests not a one-off patch, but how an agent extends its own code after new requirements.
Nick Dobos
@nickadobos
Why are there no benchmarks for code without slop? Sure, Claude scored 70% on FrontierSWE or whatever. But it also stuffed in 100 nonsense tests that no sane engineer would keep. It wouldn't pass code review. Now I have to check all of it. If a person has to review and clean up after the agent, that's a failure, not a good score.
· 26.4K views
**Previously, only the outcome was counted.** FrontierSWE v2 gives five attempts for each of 34 tasks, with a budget of 20 hours per attempt. SlopCodeBench guides the agent through 36 tasks and 196 checkpoints, while hidden tests check CLI and API behavior.
Review counts too. SlopCodeBench separately measures unnecessary code and rising complexity in already convoluted functions. In the study, agents' code was 2.3 times more verbose and 2.0 times more prone to structural erosion than 473 open-source repositories. Quality-focused prompts did not slow the degradation: a checkpoint became 12.1% more expensive, while correctness fell by 2.3 pp.
The repository installs with `uv`: after `git clone https://github.com/SprocketLab/slop-code-bench.git && cd slop-code-bench && uv sync`, you need Python 3.12+, Docker, an agent API key, 8 GB RAM, and 10 GB of disk space. A run is checked with `slop-code eval outputs/your-run-directory/`, while `slop-code metrics judge` evaluates the code with a judge model against a rubric; the first Docker image build takes 5–10 minutes.
The authors call SlopCodeBench an initial release, not a finalized benchmark.
