September 22, 2026
In-house measurements move AI products fastest: Kilpatrick advises spending more than a quarter of the time on them
A custom benchmark can be assembled in eight Kaggle CLI commands, while model runs are limited by a $5-per-day quota.
Logan Kilpatrick
@officiallogank
If you are building a product on AI, more than a quarter of your time should go toward creating benchmarks and getting model labs interested in those benchmarks. The simplest way to speed up a company’s development.
· 580.7K views
A custom benchmark has stopped being a quarter-long project. Eight terminal commands and a $5-per-day quota.
Public rankings, including our Index, answer the question of “which model is stronger overall.” A product needs a different answer: which model does not break its workflow. The same agent that writes code can now assemble that evaluation.
Before and now. On 14.01.2026, Kaggle launched Community Benchmarks: anyone can build a task set, run it, and rank Google, Anthropic, and DeepSeek models, with access free within the quota. On 4.06.2026, building moved into local environments (Antigravity, VSCode, Cursor) through the Kaggle CLI and the kaggle-benchmarks SDK. By then, the community had accumulated more than 10 000 benchmark tasks.
How to build your own. Give an agent the write-kaggle-benchmarks skill from Kaggle’s kaggle-skills repository, then describe the task in plain language: “build a task that asks the model whether 300+140=460 is correct.” The full cycle takes eight CLI steps. First `kaggle b init -y`, then a Python file with the `@kbench.task` decorator and a local run with `python task.py`. Then push the task, run it on the chosen model, `status`, `download`, and `publish`.
Runs are constrained not by time but by money. The Model Proxy quota is counted in dollars; the docs give examples of $5 per day and $100 per month, while `kaggle benchmarks quota` shows the remaining balance. Local init provides a shortened model list; the full catalog works on Kaggle infrastructure.
Outside Kaggle, there is Inspect, an open framework from the UK AI Security Institute and Meridian Labs: `pip install inspect-ai`, more than 200 ready-made evaluations, run with `inspect eval simpleqa.py --model openai/gpt-4o`. Anthropic’s evaluation docs recommend scale and automation. Answers are compared with a reference, measured for semantic similarity, or judged by another model on a scale from 1 to 5. Many auto-evaluated cases are more useful than a handful of manual ones.
On 19.09.2026, Logan Kilpatrick named the measurement the market lacks on the Sequoia podcast: Harness Bench, because no one measures how models adapt to different scaffolding.
