September 22, 2026
Testing AI agents gets easier: a set of 8 skills for evals
On September 22, 2026, `evals-skills` assembled 8 skills for evaluating a specific AI product rather than a model's general benchmark performance. In Codex, plugin v0.3.1 installs with two commands: `codex plugin marketplace add ai-evals-course/evals-skills` and `codex plugin add evals@ai-evals-course`.
Lenny Rachitsky
@lennysan
A small tip: install this new evals skill from @HamelHusain and @sh_reya. It will save you many hours and prevent plenty of mistakes. https://github.com/ai-evals-course/evals-skills
Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp took its automatic receipt collection from 35% to 83% accuracy. — @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced. — @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score. — @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost. So I asked the 🐐s of evals, @HamelHusain and @sh_reya, to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting. Read it here: https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix
· 192.5K views
Previously, teams looked at base-model benchmarks. Now the set guides users from `evals-start` through auditing, error discovery, synthetic data, judge prompts, evaluator validation, RAG, and a review interface.
Failure analysis. `error-discovery` takes JSONL, CSV, or JSON containing model responses or traces, builds a review app, and groups error types. For datasets with more than 50 records, it starts with 15–25 examples: 60–70% from clusters and 30–40% at random. When run through `codex exec`, the skill builds and checks the app, stops the server, and leaves a command for the next run.
The full set installs with `npx skills add https://github.com/ai-evals-course/evals-skills`; you can check and update it with `npx skills check` and `npx skills update`. The authors suggest starting with `evals-start`, assigning `eval-audit` in parallel across diagnostic areas, and consolidating the results into one report.
The authors released the set on March 2, 2026, then updated the installation and failure-analysis instructions on September 22.
Source
