GitHub · Agents and tools · Benchmarks and evaluation
openlayer-ai/jevals
Running evals on every trace instead of on a sample of them.
What it does
Runs agent evaluations — tool choice, groundedness, staying in scope — as typed decisions, one request per trace, with the model swappable between Jev, Kev, a local Laya or a plain chat model.
The repository's own description: “Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.”
How it works
Each eval is written as a question with an answer shape the model can produce directly, all of a trace's questions are sent together, and the answers come back as probabilities the calling code thresholds.
The repository, by the numbers
Read from the GitHub API on 2026-09-24. Stars and the last push move daily — quote them with the date, the way we do.
What we checked
- the README and the package page, read 2026-09-24
- the list of supported backends and the keys each one needs
- the licence file
What we did not check. The cost per trace and the claim that the chat-model fallback is slower and dearer. Both need a run with an API key and a real trace.