October 11, 2026
AI agents rejected Elixir PRs but accepted them in Go and Rust: the difference is in the set of checks
The Elixir port of Campfire is checked by 1,961 tests for parity with Rails, and AI reviewer agents judge its PRs differently from PRs in Go, Express and Rust. According to José Valim, the reason is the set of required checks: each project has its own.
José Valim
@josevalim
I was trying to figure out why AI agents reject PRs that speed up the Elixir version, while accepting PRs in Go, Express and Rust. It turns out each project has not only its own starting point and prompt, but also its own set of checks to pass! For example, only Elixir is explicitly required to “preserve the behavior of the pinned Rails reference.” Only it has an explicit check that we use Redis and drain its queue (so it can't be removed, unlike in other projects). And only its rule is actually followed. So people who tell you not to read the code can't use AI to set up honest benchmarks themselves. Probably because that would mean reading the code. Don't take my word for it: run your own research or ask an agent to do it for you. For example, here is what Claude Code returned: https://gist.github.com/josevalim/aecb6a602673a5dde173dd825346f749
Also extracted a separate verification project that's responsible for the benchmarking and checking of new implementations and improvements to the existing ones. The full report from the latest runs is there too. https://github.com/basecamp/once-campfire-verification/blob/main/docs/performance-review.md
· 48.2K views
José Valim wanted to understand why AI agents rejected PRs on Elixir while accepting the same kind of PRs in Go, Express and Rust. He found the cause was not in the prompts. According to him, the Elixir port of Campfire has its own checks: only it must preserve the behavior of the pinned Rails reference, and only it has an explicit check that the Redis queue gets drained. Valim published the Claude Code report on this openly; it is linked as Claude Code in the tweet.
The speed-up showed numbers. PR #4, “improve runtime throughput while preserving Rails behavior,” agreed with the agents, raised throughput at 16 connections by 19.4% for rooms, 15.6% for messages and 63.3% for the sidebar. The port itself on the rooms page served 5,350 requests per second at 16 concurrent clients, against 4,101 for Rails.
The parity check. PR #4 passed 65 parity gates against Rails and 1,941 unit tests on amd64 and ARM64 Linux. Across 12,858,383 successful responses in 192 HTTP cells there were no errors, and the Redis and Resque queues were drained.
For practitioners, the takeaway is this: an agent's result depends on which checks the project sets, not only on the prompt.
Regressions in PR #4 remain open: at 64 connections, the slowest 1% of responses in rooms became 54.4% slower, and in messages 87.0% slower.
Source