We Tested 5 Decision Models on 420 Real Requests — and the Results Flipped Our Expectations

First: What Are Decision Models?

Over the past year, a new class of AI model has emerged, commonly called Decision Models — and they differ fundamentally from conventional generative language models:

• A generative language model produces text token by token, and the application then parses that text to extract a "decision" — a fragile, costly, and slow process.

• A decision model takes in a state — text, JSON, or images — and returns a structured decision directly: a choice from a defined set (Choice), a rating on an ordered scale (Score), or a boolean probability (Noul), along with calibrated probability distributions.

This difference makes them ideal candidates for the structural tasks inside applications: routing, classification, triage, and automated verification — the places where a fast, predictable answer matters more than expressive prose.

And with the pace of releases accelerating — five new or updated decision models within just two weeks before our benchmark ran — the practical question became urgent: which of them actually deserves a place in production?

We refused to answer with impressions. We built a unified testing harness, ran it against five models through AI Gateway, and ended up with 420 real requests whose results are fully documented below.

Second: The Models Under Test

Before presenting the results, here is a documented profile of each model, drawn from its maker's official sources:

1. pplx-decider-flash-v1.2 — Perplexity

Perplexity's newest release at the time of our benchmark (published October 16, 2026). A 4-billion-parameter decision model built on Qwen3.5-4B, trained via distillation from Perplexity's larger decision models — the student learns the teachers' full probability distributions, not just their final answers. It shares the exact same API contract as the 27B model, supports text, JSON, and images, and explicitly targets high-volume classification, routing, labeling, and filtering.

• Context: supports text, JSON, and images (multimodal)

• Official pricing via AI Gateway: $0.01 per million input tokens (billing applies to input tokens only)

2. pplx-decider-v1.2 — Perplexity

[📸 Image: screenshot of the official spec card — Hugging Face]

The full-size member of the Decider family: a 27-billion-parameter decision model fine-tuned from Qwen3.8-27B. The first pplx-decider shipped on October 1, 2026 as Perplexity's first decision model, and the v1.1 update raised the company's "Decision Index" score from 56.4 to 61.56 — with a claimed lead of more than 3.5 points over Jev. It supports up to 128 named questions per request, up to 255 options per choice question, and a 256K context window.

• Context: 256K tokens, multimodal (text + images)

• Official pricing via AI Gateway: $0.02 per million input tokens

3. jev-1.14.0 — TypeSafe AI

The best-known model in this category so far, and the first that TypeSafe AI branded as a "System One Model": a model that returns typed answers (Choice / Score / Boolean) with calibrated probabilities instead of free-form text. Jev never generates prose at all — which partly explains the remarkably stable response times we observed in our results below. It supports context windows up to 262K tokens (long context is disabled by default and enabled via a dedicated header).

• Context: 262K tokens (text and JSON only — no images)

• Official pricing: $0.042 per million input tokens, with output tokens free ($0)

4. d1-ultra — Liquid AI

Liquid AI's newest release (published October 15, 2026): a multimodal decision model supporting both text and images with a 32,768-token context window. It returns calibrated answers (Noul / Choice / Score) in a single forward pass without generating a single output token. Liquid explicitly positions it against the "harder, messier" real-world cases: spotting a defect on a circuit board against its spec, or checking a damage claim against the customer's photo.

• Context: 32,768 tokens, multimodal (text + images)

• Official pricing: $0.042 per million input tokens (input-only billing)

5. laya — ConvAI Innovations

The most architecturally different model in the group: a non-autoregressive decision model that processes the entire decision in a single forward pass, with striking claimed specs: a response time of roughly 33ms on GPU, support for more than 100 languages, and a design built on RLCD and proper scoring rules. The model ships with open weights, and its context window is 8K tokens. We included it in the benchmark precisely because of those claims — and the result, as you will see, was among our biggest surprises.

• Context: 8K tokens

• Pricing: unlike the others, it is charged per decision (roughly $0.0004 per request according to its published pricing page) rather than per token, and it does not appear in AI Gateway's price list

Documentation note: all specs and prices above come from each company's official model pages and AI Gateway's pricing page as of the benchmark date (October 17, 2026), and are subject to change.

Third: Benchmark Methodology

We designed the experiment to be reproducible and challengeable:

1. 14 unified scenarios spanning six distinct capability categories, presented verbatim — same wording — to every model:

   • Logic: chain inferences such as "Layla is taller than Sara, and Sara is taller than Noura — who is the shortest?"

   • Numbers: practical comparisons and calculations, such as identifying the cheaper per-unit offer

   • Classification: categorizing customer-support messages and news-headline tone

   • Triage: prioritizing emergency-room cases

   • Causal reasoning: inferring the most likely explanation for an economic anomaly

   • Multiple constraints: scheduling a meeting for three people with conflicting calendars, and selecting a product that satisfies simultaneous requirements — the hardest category in the set

2. Two versions of every scenario: Arabic and English — to isolate the effect of language on decision quality specifically.

3. Three repetitions of every test — we had previously observed real variance in these models' behavior across sessions; repetition is a methodological necessity, not a luxury.

4. The final count: 5 models × 14 scenarios × 2 languages × 3 repetitions = 420 requests.

5. Technical execution: all requests went through experimental_decide from the AI SDK — the official interface for decision models — with models routed via AI Gateway. Decisions came back structured in 100% of successful requests, with no manual text parsing that could distort the results. Request timeout was 30 seconds, and any failure was logged without halting the run.

The metrics we used:

• Accuracy (Hit@1): the share of correct answers. Since every scenario in our suite has exactly one correct answer, this is the most honest metric — we deliberately excluded the conventional MRR metric because it collapses into a linear function of Hit@1 for single-answer choice questions.

• Latency: mean, median (p50), and 95th percentile (p95) — because the mean alone misleads when the distribution is skewed.

• Reliability: how many requests failed outright.

• Cost: computed from the actual reported tokens in every request and each provider's official prices.

Fourth: The Results — Main Table

* Laya is priced per decision rather than per token and is not listed in AI Gateway's price table — we excluded it from the cost comparison.

The Analysis: Three Entirely Different Stories

Story one — perfect accuracy and its time price: pplx-decider-flash scored 100% without a single exception: 84 out of 84, in both languages, across all six categories. But the latency distribution reveals another face: p95 hit 2932ms and the slowest request 7528ms — meaning roughly one in every twenty requests takes over three seconds. Its best-in-class accuracy is paid for in the tail of its latency distribution.

Story two — silent stability: Jev was not the most accurate (99% — one wrong answer in an English test out of 84), but its latency distribution is the most striking in the entire experiment: the slowest request across 84 requests was 612ms. No meaningful tail, no spikes, no outages. That consistency — not raw speed — is what makes it the first production candidate for live decision systems, where strict deadlines matter more than ideal averages.

Story three — the collapse: laya's performance was not merely weak; it sat exactly at the random-guessing floor (25% with four options per question). The per-language and per-category breakdown below makes the picture even darker.

Fifth: Arabic vs. English

This section carries the two most important findings of the benchmark:

The good news: the top three models were completely unaffected by Arabic — in fact, Jev and d1-ultra scored 100% in Arabic, with their only mistakes in English. The common claim that "Arabic is a hurdle for AI models" simply does not hold, at least in the domain of structured decisions, where fully Arabic context and questions were handled flawlessly.

The bad news: laya collapsed specifically in Arabic — just 7%, a single correct answer out of fourteen. More striking still, this directly contradicts the company's official claim of supporting "more than 100 languages." One plausible explanation is that the multilingual claim is built on the company's own benchmarks and does not cover Arabic in sufficient depth, or that Arabic text handling in a non-autoregressive architecture needs further work. Either way, the practical conclusion is clear: do not rely on laya for any Arabic decision.

Sixth: Accuracy by Capability Category

This table exposed a second methodological surprise: pplx-decider (the full 27B version) collapsed specifically in the logic category — 29% — while its own distilled 4B sibling scored 100% on the exact same questions. The failure is not in general capability: the model is competent in numbers, classification, triage, and causal reasoning. The flaw is confined to serial logical inference — the one category that genuinely distinguishes "deciding" from "classifying." The practical takeaway: never assume bigger means smarter; always test both sizes.

As for laya, it scored an absolute zero in the causal-reasoning and multiple-constraints categories — the two categories that require assembling information from multiple parts of the state before issuing a judgment, rather than simple pattern matching.

Seventh: Operational Reliability

12 of 84 requests to pplx-decider (14%) failed outright with a Service temporarily unavailable error from the provider, scattered randomly across scenarios — a service-stability problem, not a weakness on specific questions. The other four models logged zero failures across 84 requests each. The lesson here is simple: a model that fails one in every seven requests is unfit for production no matter its theoretical accuracy, because every failure means fallback logic you must build and maintain.

Eighth: Cost — Actual Accounting, Not Sticker Prices

We computed cost from the tokens actually reported in every request, at each provider's official rates:

Three observations deserve attention:

1. The entire experiment — 420 real requests — cost less than half a cent. That is the real value proposition of decision models: a trustworthy structured decision for a fraction of a cent, with no full text generation to pay for and parse.

2. The price of a token is not the cost of a decision. Jev charges four times more per token than flash, yet it consumed 2.5× more input tokens on identical prompts — ending up the most expensive of the priced models, despite a quality gap that does not justify that premium. When choosing a decision model, compute the actual per-decision cost, not the per-million-token sticker price.

3. A methodological caveat: the large gap in reported tokens across providers (36,669 for Jev vs. 14,541 for flash on identical prompts) may reflect differing token-accounting methodologies rather than genuinely different consumption — which is why we caution against generalizing these cost figures outside their context.

Ninth: The Lessons

1. "Newest" does not mean "best for production." Flash achieved the highest accuracy (100%) but paid for it in the tail of its latency distribution (p95 six times higher than Jev's). If your decision must return within half a second to a waiting user, the newest model is emphatically not the answer.

2. Latency stability is an underrated advantage. Compare "slowest request": 612ms for Jev vs. 7528ms for flash. In live routing and triage systems, it is the tail that takes your service down, not the mean.

3. The bigger model can lose to the smaller one. pplx-decider-27B's logic collapse against its own distilled 4B sibling is a lesson in never assuming monotonic improvement with model size.

4. Arabic is not a barrier in decision tasks. The top three models performed in Arabic equal to or better than in English — a result worth building Arabic-market products on.

5. The gap inside the category is wider than the gap between categories. The difference between flash/jev/d1-ultra (98–100%) and laya (25%) is larger than the gap between a good decision model and a conventional language model. The "decision model" category is still in its natural-selection phase, and marketing claims (like multilingual support) need independent testing, always.

6. Unified infrastructure saves real effort. AI Gateway let us run the entire benchmark across five different providers by changing a single line (the model ID), with no separate keys or provider-specific code — without it, this experiment would have taken several times longer.

Tenth: The Recommendation

As for us, Jev remains our production choice — not because it is theoretically the most accurate, but because it is the only one you never fear at peak hours: our slowest request was 612ms, zero failures, and just one decision away from "perfect."

Eleventh: The Limits of This Benchmark — In Full Honesty

So you can weigh our results appropriately:

• 14 scenarios is a small sample: large enough to expose radical gaps (laya's and pplx-decider's collapses, for instance), but a 98% vs. 99% difference could flip in another run.

• Executed from a single machine over the internet: latencies are affected by the network; running the benchmark from a server close to the providers would yield lower numbers.

• Flash's latency tail may be partly transient network pressure — but it recurred often enough to be taken seriously.

• Prices are from AI Gateway at benchmark time and subject to change, and differing token-accounting methodologies across providers limit the precision of absolute cost comparisons.

• Our methodology is fully published: the scenarios and code are open, and anyone can reproduce the benchmark and challenge our results — which is, in our view, the hallmark of a trustworthy technical report.

Conclusion

Decision models are not a passing wave; they genuinely redefine the relationship between applications and AI: from "a text generator we hope understands" to "a structured decider that returns typed answers with calibrated probabilities." But the market is still young: between a model that scores 100% and one that scores 25% — and between a claim of 100-language support and a real collapse in Arabic — independent, numbers-driven testing remains the only tiebreaker.