AI engineering

Jev Benchmarks: Cheaper Decisions, With What Tradeoff?

TypeSafe reports similar agreement at lower cost in one comparison, and better agreement from another model. Keep the two comparisons separate.

Listen to the audio analysis

Create your own

Written for builders and creators.

Most AI automation today is built on a fundamental mismatch. We're trying to force conversational models, which were designed to chat and be helpful, to act like deterministic software.

Editor's note

Why this matters now

Jev is TypeSafe's model for structured decisions, such as classifying a request or assessing an invoice. Anthony Maio's September 2026 account includes a useful comparison for engineers considering it: Jev reached 67.8% agreement across 4 vendor-designed workflows, while a general model reached 67.9% at much higher reported cost and latency.

A different model reached 74.1%, but the account does not supply its cost. Those are 2 separate comparisons. Together they show why a decision-model benchmark needs quality, cost and task detail beside each other before it can guide a replacement.

The source

What it says

Distilled from the original. The notes above and below are the editor's own.

What Was Compared

TypeSafe's internal evaluation covered security response, agent-trace observability, invoice processing and customer service. The reference answers were generated using other language models. The resulting scores measure agreement with those labels, rather than independently verified real-world correctness.

Maio reports these results:

Model Label in the AccountAgreementCost per CaseLatency per Case
Jev67.8%$0.00040.4 seconds
GPT “Terra”67.9%$0.030410.1 seconds
GPT “Sol”74.1%Not specifiedNot specified

The displayed costs imply a ratio of about 76 to 1, consistent with the account's rounded 75-fold claim. The displayed latency ratio is about 25 to 1. Neither ratio describes the comparison with “Sol,” whose cost and latency are missing here. The labels are those used in the source account, not a current model-buying guide.

The Average Hides a Larger Invoice Gap

Jev's overall agreement was 6.3 percentage points below the strongest model in the comparison. On invoice processing, Jev reached 61.8%, against 79.1% for that model: a 17.3-percentage-point gap.

That is enough to change the adoption question. A team routing low-cost support requests might accept a different error profile from a team making decisions about invoices. The overall average does not reveal which disagreements would be expensive, recoverable or harmless in either system.

The evaluation also cannot establish a broad tradeoff between speed and “raw intelligence.” It covers 4 particular workflows, with their chosen questions and labels. A lower agreement score could reflect genuine mistakes, differences from the label-generating models, or both. Resolving that requires examining the cases.

Why a Decision Model Could Reduce Overhead

Jev replaces open-ended generation with typed questions. Its Choice primitive selects a declared option, Score evaluates ordered descriptions, and Noul estimates the probability of a proposition. Those interfaces let application code consume the decision directly.

This makes the comparison most relevant when the existing generative call is being used to return a small judgment. It is less relevant when the application needs a plan, an explanation or new text. The Jev introduction describes that boundary in more detail.

Constrained output also addresses only one kind of reliability. A valid category can still be the wrong category. A schema needs to cover the decisions the application actually encounters, including an uncertainty route where appropriate. Otherwise the model can return a confident choice from an incomplete set.

What the Vendor Evaluation Leaves Open

TypeSafe designed the workflows and used model-generated reference labels. Independent assessment would help establish whether the disagreements represent errors and whether the selected cases resemble a reader's intended deployment.

The account describes reinforcement learning for calibrated decisions, but does not disclose the full training procedure or the statistic behind every confidence field. A reported probability is useful only to the extent that it predicts outcomes on the relevant cases. The gateway article explains how that affects escalation thresholds.

For a meaningful comparison, the models need to receive the same inputs and decision options. Costs should include unsuccessful requests and any escalation needed to reach an acceptable result. Measuring only the first cheap answer can conceal the work required after it.

Editor's note

What to do with this

The supported conclusion is specific: TypeSafe reports a much cheaper and faster result at nearly the same label agreement as one comparator. Another comparator achieved higher agreement, especially on invoices, without a reported price here. Jev is therefore a candidate for bounded decisions whose quality requirements can be measured. The published average alone cannot identify which production step should move to it, or what the total savings would be once errors and escalations are included.

The original

Jev: the language model that won't talk

Anthony Maio · 16 September 2026

  • AI engineering

    What Is Jev? A Model Built to Make Typed Decisions

    Jev returns choices and probabilities for software to act on. Its decision interface explains both the speed claims and the jobs it cannot take over.

    4 min read7 min listen

  • AI engineering

    When Can a Jev Confidence Score Drive a Decision?

    Jev is trained to return calibrated decisions. Learn how probabilities differ from confidence, and what a gateway still has to decide for itself.

    4 min read5 min listen

All posts