Repository · read 2026-10-17
muse-spark-open-bench
Tests whether the model catches the bug a change actually shipped. It reviews 50 merged pull requests from popular open-source projects, 25 of which were later reverted or fixed, and grades each review against the maintainers' own fix rather than against a rubric written for the benchmark.
The repository → 0 stars · TypeScript
| Field | Value |
|---|---|
| Repository | Studio1-OSS/muse-spark-open-bench 0 stars · read 2026-10-17 |
| Language | TypeScript Last push 2026-10-16 |
| Licence | MIT Repository licence; the model weights carry their own. |
| Stack | Node.js 24, Nebius Token Factory endpoint for Muse Spark Open What it runs on. |
| Best for | Readers who want a second opinion on the coding claims in Meta's launch table, from a harness the vendor did not write. A reader, not a benchmark. |
How it works
Each review starts from a real repository and lets the model search and read files before writing a review. A catch only counts if it names the cause specifically enough to turn into a failing test, and catches are confirmed by running the code. Ten visual cases add a user's screenshot of the broken interface.
What we checked
- The results table and its Wilson interval
- The cost and latency lines
- The list of repositories tested and the visual-bug section
- The note that runs require a Nebius Token Factory key and cost about $20
What we did not check
We did not rerun the benchmark or verify its grading code; the numbers are the authors' saved run.
Our note
An independent code-review benchmark: 50 real merged pull requests, 25 of which were later reverted or fixed, run through Muse Spark Open with every catch checked by running the code.
github.com/Studio1-OSS/muse-spark-open-bench Verified 2026-10-17 Stars 0 at read time