AI engineering
Jev Benchmarks: Cheaper Decisions, With What Tradeoff?
TypeSafe reports similar agreement at lower cost in one comparison, and better agreement from another model. Keep the two comparisons separate.
Listen to the audio analysis
Create your ownWritten for builders and creators.
Most AI automation today is built on a fundamental mismatch. We're trying to force conversational models, which were designed to chat and be helpful, to act like deterministic software.
Editor's note
Why this matters now
Jev is TypeSafe's model for structured decisions, such as classifying a request or assessing an invoice. Anthony Maio's September 2026 account includes a useful comparison for engineers considering it: Jev reached 67.8% agreement across 4 vendor-designed workflows, while a general model reached 67.9% at much higher reported cost and latency.
A different model reached 74.1%, but the account does not supply its cost. Those are 2 separate comparisons. Together they show why a decision-model benchmark needs quality, cost and task detail beside each other before it can guide a replacement.
The source
What it says
Distilled from the original. The notes above and below are the editor's own.
What Was Compared
TypeSafe's internal evaluation covered security response, agent-trace observability, invoice processing and customer service. The reference answers were generated using other language models. The resulting scores measure agreement with those labels, rather than independently verified real-world correctness.
Maio reports these results:
| Model Label in the Account | Agreement | Cost per Case | Latency per Case |
|---|---|---|---|
| Jev | 67.8% | $0.0004 | 0.4 seconds |
| GPT “Terra” | 67.9% | $0.0304 | 10.1 seconds |
| GPT “Sol” | 74.1% | Not specified | Not specified |
The displayed costs imply a ratio of about 76 to 1, consistent with the account's rounded 75-fold claim. The displayed latency ratio is about 25 to 1. Neither ratio describes the comparison with “Sol,” whose cost and latency are missing here. The labels are those used in the source account, not a current model-buying guide.
The Average Hides a Larger Invoice Gap
Jev's overall agreement was 6.3 percentage points below the strongest model in the comparison. On invoice processing, Jev reached 61.8%, against 79.1% for that model: a 17.3-percentage-point gap.
That is enough to change the adoption question. A team routing low-cost support requests might accept a different error profile from a team making decisions about invoices. The overall average does not reveal which disagreements would be expensive, recoverable or harmless in either system.
The evaluation also cannot establish a broad tradeoff between speed and “raw intelligence.” It covers 4 particular workflows, with their chosen questions and labels. A lower agreement score could reflect genuine mistakes, differences from the label-generating models, or both. Resolving that requires examining the cases.
Why a Decision Model Could Reduce Overhead
Jev replaces open-ended generation with typed questions. Its Choice primitive selects a declared option, Score evaluates ordered descriptions, and Noul estimates the probability of a proposition. Those interfaces let application code consume the decision directly.
This makes the comparison most relevant when the existing generative call is being used to return a small judgment. It is less relevant when the application needs a plan, an explanation or new text. The Jev introduction describes that boundary in more detail.
Constrained output also addresses only one kind of reliability. A valid category can still be the wrong category. A schema needs to cover the decisions the application actually encounters, including an uncertainty route where appropriate. Otherwise the model can return a confident choice from an incomplete set.
What the Vendor Evaluation Leaves Open
TypeSafe designed the workflows and used model-generated reference labels. Independent assessment would help establish whether the disagreements represent errors and whether the selected cases resemble a reader's intended deployment.
The account describes reinforcement learning for calibrated decisions, but does not disclose the full training procedure or the statistic behind every confidence field. A reported probability is useful only to the extent that it predicts outcomes on the relevant cases. The gateway article explains how that affects escalation thresholds.
For a meaningful comparison, the models need to receive the same inputs and decision options. Costs should include unsuccessful requests and any escalation needed to reach an acceptable result. Measuring only the first cheap answer can conceal the work required after it.
Editor's note
What to do with this
The supported conclusion is specific: TypeSafe reports a much cheaper and faster result at nearly the same label agreement as one comparator. Another comparator achieved higher agreement, especially on invoices, without a reported price here. Jev is therefore a candidate for bounded decisions whose quality requirements can be measured. The published average alone cannot identify which production step should move to it, or what the total savings would be once errors and escalations are included.
The original
Jev: the language model that won't talk
Anthony Maio · 16 September 2026
Read next
AI engineering
What Is Jev? A Model Built to Make Typed Decisions
Jev returns choices and probabilities for software to act on. Its decision interface explains both the speed claims and the jobs it cannot take over.
4 min read7 min listen
AI engineering
When Can a Jev Confidence Score Drive a Decision?
Jev is trained to return calibrated decisions. Learn how probabilities differ from confidence, and what a gateway still has to decide for itself.
4 min read5 min listen