September 21, 2026
Code-to-video comes out lifeless from models: the leader in Code2Video Bench scored 0.32 out of 1
HeyGen, together with Google DeepMind and Kaggle, released Code2Video Bench: 16 models write code for videos, and none has caught up with humans.
HeyGen
@heygen
Models can write code. Whether it renders into a video worth watching is a separate question. That is exactly what the Code2Video bench measures. Built with Google DeepMind and @kaggle.
if you tried hyperframes, you would know the agentic video stack is being built on code-gen but SWE-bench code v.s. code-to-video that feels alive are two different things we built the Code2Video Bench, in collab w/ Google DeepMind & @Kaggle frontier labs can finally get good at agentic video tasks
· 11.4K views
None of the 16 models produced a human-level video. The best result in the evaluation: 0,32 out of one.
HeyGen published Code2Video Bench on 18.09.2026 together with Google DeepMind and Kaggle. The benchmark tests how a model writes HTML and React that are then rendered into motion graphics.
Tickets used to be the metric. SWE-bench measures whether a model closes an issue in a repository. Here, the question is whether the result looks good: 168 human reference compositions are broken down into eight beats of a product launch, from hook to branded outro.
The gap from humans is so large that win rate loses meaning, so the Kaggle leaderboard from 21.09.2026 uses Elo. GPT-5.5 leads with 1574,5. There is no clear winner: the top-4 are within 12 Elo of one another with a confidence interval of about ±13, while open weights (qwen3.8-max, kimi-k3, and two others) trail first place by less than 50 Elo.
Layout is where it breaks. Models have nearly closed the gap on following the brief: they cluster tightly on the Prompt-Intent axis. They diverge on Composition and Craft: timing between beats drifts, text overlaps itself, and layouts come out small and cautious.
You can run your own video the same way the authors did: `npx skills add heygen-com/hyperframes`, with Node.js 22+ and FFmpeg required, then `npx hyperframes init`, `preview`, and `render`. It is not evaluated by a standard VLM, but by a HeyGen-trained judge model across five axes: it agrees with humans 82% of the time, versus 75% for VLM.
Kaggle will take the leaderboard further and promises to open a run of its own model with the same judge.
Source
