GitHub · Runtime ports · Serving and API
lkarlslund/laya.cpp
Deployments that need a native binary — Windows, a GPU, or a machine without Python.
What it does
Native C++ inference for all three checkpoints, including tokenisation and JSON output without Python, plus a Jev-compatible server with automatic batching. Prebuilt CUDA and Vulkan executables ship for Windows and Linux, with a Core ML build for Apple silicon.
The repository's own description: “C++ inference for Laya typed decisions - supports CUDA, Vulkan, Core ML, CPU”
How it works
The weights are converted to ggml and dispatched to whichever backend the machine has; the server groups concurrent requests so a busy agent loop pays for the forward pass once instead of once per question.
The repository, by the numbers
Read from the GitHub API on 2026-09-24. Stars and the last push move daily — quote them with the date, the way we do.
What we checked
- the README, read 2026-09-24
- the paired comparison table and what it says it compared (250 fixed questions, three models, batch sizes 1 to 8)
- the licence file
What we did not check. The performance ratios in that table. They are the author's, measured against his own Python baseline, and we ran neither side.