October 13, 2026
Classification can move off language models: Jev 1.13 is 4.4x cheaper than Claude Haiku 5.5
Jev 1.13 completes a task in 0.13 seconds, Claude Haiku 5.5 takes 1.4: Artificial Analysis has built the first index of such models.
Artificial Analysis
@artificialanlys
Introducing the Artificial Analysis Decision Index: we measure decision models on classification, tool calling, knowledge and reasoning. Jev leads among decision models; language models like Claude Haiku 5.5 are still more accurate, but they cost much more and respond more slowly. Decision models output structured answers or classifications with probabilities: they respond faster and cost less than language models. Artificial Analysis Decision Index v1.0 compares the capabilities, price and speed of decision models, and also compares them with language models. Benchmarks in Artificial Analysis Decision Index v1.0: ➤ AA-Omniscience-Decision: a held-out set of 500 questions adapted from our proprietary AA-Omniscience benchmark. Measures decision models' knowledge and their tendency to hallucinate ➤ HLE-MultiChoice: 513 text-only multiple-choice questions from Humanity's Last Exam, testing decision models' ability to reason ➤ When2Call: a tool-calling benchmark from NVIDIA. Tests whether a model correctly decides to call a tool, ask a follow-up question, decline to answer or answer directly. These are the capabilities needed in agentic workflows and tool routing ➤ WildGuardTest: a safety benchmark that evaluates classification of harmful user prompts, assistant refusals and harmful responses. We use a subset of 835 rows out of the original 1,725, those where all three human annotators agreed on all three labels, and exclude questions that models consistently refuse to answer Key results: ➤ Jev 1.13 leads the Decision Index and sits on the capability vs. price Pareto frontier: Jev 1.13 from @typesafeai scores 58 on the Artificial Analysis Decision Index, ahead of GLiDE (54) from @fastinoAI and Microsoft-Decision-1 (54) from @Microsoft. Decider v1.1 27B from @perplexity_ai has the lowest price per 1M tasks, $17; Microsoft-Decision-1 ($31) and Jev 1.13 ($42) are also on the Pareto frontier. Celeris-1 Decision has the shortest time per task, 0.06 seconds: more than 2x faster than Jev 1.13 (0.13 seconds per task) ➤ Decision models are poorly calibrated on knowledge and reasoning tasks: at almost every probability level of the chosen answer, accuracy for all decision models falls below that probability. The gap is largest on AA-Omniscience-Decision and HLE-MultiChoice, while on WildGuardTest (classification) and When2Call (tool calling) most models are reasonably well calibrated ➤ Decision models clearly beat language models on speed and price but lag on knowledge and reasoning: on classification and tool calling, the best decision models perform on par with language models while being much faster and cheaper. On knowledge and reasoning, however, language models are far ahead. Claude Haiku 5.5 (Low) scores 61 on the Decision Index, 3 points above Jev 1.13, but takes 1.4 seconds per task versus 0.13 (11x slower) and costs $184 per 1M tasks versus $42 for Jev 1.13 (4.4x more expensive) We will keep developing the Artificial Analysis Decision Index to make it clearer for developers and companies when and how to use decision models.
· 46.2K views
A million classifications cost $42 with Jev 1.13 and $184 with Claude Haiku 5.5.
On Oct 13, Artificial Analysis released Decision Index v1.0, the first shared benchmark of models that return a label with a probability instead of text (they are called decision models). An agent pipeline makes many such calls: call a tool or ask for clarification, harmful request or ordinary one, which branch to route it to.
What was measured. The index combines four benchmarks: knowledge and hallucinations on 500 closed questions from AA-Omniscience-Decision, reasoning on 513 multiple-choice questions from Humanity's Last Exam, tool selection on NVIDIA's When2Call, and harmful request classification on WildGuardTest. In WildGuardTest, Artificial Analysis kept 835 rows out of 1,725, those where all three annotators agreed on all three labels.
The index is led by Jev 1.13 from typesafeai with 58 points. Next come GLiDE from Fastino and Microsoft-Decision-1, both at 54. The cheapest in the test was Decider v1.1 27B from Perplexity, at $17 per million tasks. The fastest is Celeris-1 Decision, at 0.06 seconds per task, half the time of Jev 1.13.
**Language models are more accurate but pricier.** Claude Haiku 5.5 in Low mode scores 61, 3 points above Jev 1.13. It takes 1.4 seconds per task versus 0.13 and charges $184 per million tasks versus $42. On classification and tool selection, the best decision models match language models, but they trail on knowledge and reasoning.
Until now, every such decision in a pipeline went to a language model, paid for in tokens and seconds. Now the class has its own benchmarked lineup, showing where a cheap model gets the job done and where it will get it wrong.
Probabilities run high. At almost every confidence level, accuracy came out lower than stated: the model puts 0.9 next to an answer but is right less often. The gap is widest on knowledge and reasoning. On classification and tool selection, the numbers hold up.
Artificial Analysis promises to keep developing the index, and the next versions will show whether these models close the gap on knowledge.