September 20, 2026
A cheap model can serve as an agent judge: Jev matched human labels 100% of the time
LangChain ran 500 checks: Jev matched humans in 100% of cases and cost $0.34 versus $28.17 for Claude Sonnet 4.6.
Harrison Chase
@hwchase17
“jev as a judge” cheap and fast semantic verifiers! they work great for evals, especially online ones where you need to assess a TON of traces
We tested Jev against LLM judges on accuracy, repeatability, latency, and cost to see whether System One models could offer a new approach to agent evaluation. https://x.com/i/article/2101448785255907328
· 77.2K views
The same evaluation run cost $0.34 and $28.17. The difference was not the agent, but the system judging its answers.
LangChain engineers Daniel Shi and Sean Roche took five responses from a weather agent and ran each one 100 times through four judges: 500 decisions per judge. Human annotations served as the ground truth.
The judge agreed with humans. Jev matched the annotations in 100% of cases, Claude Sonnet 4.6 in 80,0%, with both GPT-5.6 models falling between them. A judge with 80% accuracy gets every fifth decision on a trace wrong.
Jev also holds its decision steady from run to run: the variance in its scores was 92 times lower than Claude Sonnet 4.6's.
Previously, traces were judged by a large model: Claude Sonnet 4.6 spent 2,8 seconds on each one and charged $0.05634 per call. Jev responds in 0,44 seconds and charges $0.00035.
Jev does not generate text; it makes a decision. On OpenRouter, input tokens cost $0.042 per million, while output tokens are not billed at all. The model has a 32 000-token context window.
How to connect it. Get a key in the console at console.typesafe.ai/keys, then install the SDK with `pip install typesafe-sdk` (Python 3.10 or later required) or `npm install typesafe-sdk`. Requests go to api.typesafe.ai/v1/systemone, the default model is jev-latest, and questions come in three types: yes/no, multiple choice, or scoring.
With LangChain, the path is shorter: `pip install langchain-typesafe`, put the key in the TYPESAFE_API_KEY variable, and the TypeSafeClassifier class works like a standard Runnable, while Jev decisions appear in LangSmith alongside tokens and spend. Braintrust added Jev as a scorer on 18.09. The benchmark itself can be reproduced from the open jev-as-a-judge repository.
You cannot run Jev yourself: TypeSafe has published neither weights nor parameter count, and the model is available only through a hosted API. The company published its own figures when announcing System One on 15.09; LangChain later ran the benchmark on agent responses.
The next signal will come from independent benchmarks: so far, only TypeSafe itself and LangChain have published figures for Jev.
Source
