Laya vs Jev

First, the table with two parents

The comparison most people have seen is Laya's, and it is worth reading carefully because its two columns have different owners. Every Laya figure is the publisher's measurement. Every Jev figure is quoted from third parties — the card says in the line above the table that it never had TypeSafe API access, which is unusually honest and also means the table is two tables stapled together.

Two rows deserve to be read together. Laya's 0.081 ECE against Jev's 0.246 is the calibration claim, and the card is explicit that it comes from fitting a temperature per question type and option count — from 0.466 before fitting. And the row where Jev wins by 24 points is the one the card explains with a mechanism rather than a shrug: options share a fixed token budget, so 77 labels get three or four tokens each.

The same thirteen cases, sent to both

We already had an adversarial suite written for Jev — an option that was never offered, instructions hidden inside the state, a forged system line, near-synonym options, a single option, an empty state, ten thousand characters of logs, numeric keys, punctuation in the question name. It is a file in the repository, and the client that sends it is the one we send to TypeSafe. For Laya we pointed that client at the local server instead, and the table below is what came back on both sides, in the same order.

Eleven agreements out of thirteen is the result we did not expect. These are not variations of one model: Jev is a hosted service from a funded lab and Laya is 421M parameters trained by a community project, and on deliberately strange input they land on the same option. The two cases where they part are the two where the state does not really fit any option — a request to cancel an account against a four-team taxonomy that cannot cancel anything, and an empty string. Jev picked a plausible team and, in the empty case, was confident about it (0.99). Laya picked `returns` both times at 0.083 and 0.035 — the two lowest confidences in the run.

Which of those behaviours is better depends on what you do with the number. A system that routes on the label alone prefers Jev's willingness to answer. A system that escalates below a confidence floor gets a cleaner signal from Laya, which effectively says "I have nothing" where Jev says "technical, 0.99".

Twenty identical calls

This is the test that separates the two models most cleanly, and it is the cheapest one to run: send the same request twenty times and watch what moves. It is also the test that matters most in production, because a router tuned on one sample assumes the sample was the answer.

A model that returns the same four decimals twenty times is arithmetically deterministic — same weights, same input, same float ops. Jev's spread is not sloppiness either; it is a hosted service in front of a model that was never going to be bit-exact. But the consequence is asymmetric. With Jev you cannot read one confidence and decide it is "the" number: 0.38 and 0.58 sit on opposite sides of most escalation thresholds, and the request that produced one produced the other eleven seconds later.

The caveat belongs next to the claim: our Jev run was in September 2026 against the hosted endpoint and we have only one state for it. Twenty calls on one ticket is evidence that variance exists there, not a measurement of how much.

The latency numbers are not one measurement

The most repeated sentence in this comparison is "Laya is 7.8× faster". It is worth taking apart, because one side of it is a GPU with the weights already in memory and the other side is somebody else's API across the public internet.

Read down the middle column and the comparison dissolves: 32.8 ms and 0.783 s are not two points on one axis, they are three different things — silicon, a network and a batch. What a reader can actually plan with is the shape: on a CPU with no GPU, a question costs about a fifth of a second once you batch, and a single unbatched call costs a third of a second. If that is fast enough for your loop, you do not care what a T4 does; if it is not, you need a GPU and the publisher's 32.8 ms becomes the number you are buying.

The one place they behave differently in kind

Laya's `noul` — the yes/no primitive — has a documented failure mode that we reproduced on the first try. Asked "Is this review positive?" about five reviews, three of them glowing, the English checkpoint returned 0.000 at confidence 1.0 every time. Not uncertain: certain, and wrong, five for five.

The model card names this (#156), scopes it to the English checkpoint, and prescribes the workaround: ask the same thing as a two-option `choice` with neutral keys. We ran that too, and it works — all five came back on the right side, with P(yes) between 0.66 and 0.78 for the positive three and 0.004 and 0.0005 for the negative two. The multilingual checkpoint, which the router selects for non-Latin scripts, got two of three right on the same question and made its one mistake at 0.915 confidence.

Jev's equivalent does not have this failure in our runs: its `noul` returned 0.98 on the urgency question and was the most stable signal in the twenty-call test. What Jev has instead is the confidence spread above. Two different ways of being hard to trust, and the right response to both is the same: gate on your own data, not on a blog post.

What this page cannot tell you

It cannot declare a winner, and it will not: thirteen cases and forty ticket labels are enough to find edges and not enough to rank. It cannot compare accuracy on a task you care about, because neither model was trained on your task and both are meant to be fine-tuned before that comparison means anything. It cannot verify the publisher's benchmark rows for Jev, because those were never measured by the party publishing them. And it cannot tell you which is cheaper — one is free at the point of use and costs electricity, the other is $0.042 per million input tokens and costs nothing when idle, and which is cheaper depends on a utilisation curve neither of us can guess from here.