Running Laya yourself

What it takes

The install detail is not pedantry. `laya` depends on torch, and a plain `pip install` pulls the CUDA build — 2.7 GB of wheels, on a machine that will run this on its CPU. Installing torch from the CPU wheel index first takes the download to a fifth of that and the model does not notice the difference.

The loading time we quote is the current one: the card says the checkpoint build dropped from about 22 s to about 2 s because it no longer initialises throwaway random weights. The 4–6 s we measured is the wall clock including a cold file cache on a laptop; the builds after the first were faster.

What a question costs on a CPU

Three shapes of call, five repeats each, on the same ticket-like state. Batched questions are the reason this is usable at all: the per-question cost falls by a third at ten questions and then stays flat, so the thing to optimise is how many questions you ask per forward pass, not how many calls you make.

For scale: the hosted model we have measured from this machine took 783 ms median for the same shape of call, over the network. That is not a speed comparison — one number has a round trip inside it and the other does not — but it is the honest frame for "is local fast enough": a third of a second locally against about four fifths of a second remotely, before you count the GPU you would need to reach the publisher's 32.8 ms.

The budget nobody reads until it truncates

Options and state share one window. The model card puts it plainly, and it is the single most useful line on it for anyone integrating this:

A long ticket therefore gets truncated before the number you configured suggests, and the model does not refuse — it answers from whatever survived. Our ten-thousand-character case came back with a label at 0.0199 confidence, which is the shape of the right behaviour: cut off, but not certain about it.

Sharp edge one: the temperature the library will not use

Laya's calibration story is a fitted temperature per question type and option count — without it the mean ECE is 0.466, with it 0.081. Those numbers ship in the checkpoint's own config, and one of them is outside the range the library accepts:

The warning is verbatim from the library: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.

A temperature below 1 multiplies the logits, so 0.1006 would sharpen a distribution roughly tenfold and 0.5 still doubles it. That is measurable. On thirty candidate intents — six unmistakable tickets, five option-set sizes — the published confidence was 0.96 on average where the model's own untempered distribution gives 0.89:

Notice the direction at four options: there the temperature is 1.76, it softens the distribution, and published confidence is lower than the raw one (0.69 against 0.86). At eleven and above the fitted value sharpens instead. Calibration should be allowed to do both; the problem is narrower and more specific than "the numbers are wrong" — the value for the largest bucket is not the fitted one, and the library says so out loud.

What to do about it: fit your own temperature on your own data, which the card recommends anyway, and set `agent.temperature_by_options` yourself. Do not assume the calibration in the artifact is the one running in your process.

Sharp edge two: the server routes, the library does not

If you load a checkpoint directly you get that checkpoint. If you run `laya-serve`, you get whatever the router picks after looking at the script of the state — and for a non-Latin state it will not be the English one. The router says why in the response:

“non-Latin script (han, 100% of letters); the English checkpoint cannot read it”

This is a feature — the card's own numbers show the English checkpoint collapses on non-Latin script, and the router is the fix. It is also a trap for anyone who benchmarks the checkpoint and deploys the server, or the other way round. Two deployments of "Laya" can be two different models.

What did not reproduce

The card attributes the high-cardinality weakness to the fixed per-option token budget, so we tested that directly: twenty options, the same discriminative words in every label, and a neutral sentence appended to all of them. If the budget is the binding constraint, accuracy should have fallen. It moved one item out of six — 5 of 6 with short labels against 4 of 6 with 27-word labels.

One item on six tickets is noise. Reproducing the collapse properly needs the card's own benchmark at 77 labels rather than a stranger's thirty, and we are saying that rather than reporting the coin flip as a result. What we can say is the narrower thing: at 30 labels our six tickets did not collapse, they answered 5 of 6 — and the confidence climbed with the option count, which is the number to watch rather than the accuracy.

What this page cannot tell you

It cannot tell you how Laya behaves on a GPU, on Apple silicon, or under load — one laptop with nothing else running is the whole of our evidence. It cannot tell you whether 0.21 s per question is fast enough for your loop, because that depends on the loop. And it cannot tell you the memory footprint of a long-running service: we measured a process that ran for minutes, not days, and the router keeps two checkpoints resident by default.