Repository · read 2026-10-17
mlx-moe-ssd-streaming
Runs a mixture-of-experts model larger than RAM on an Apple laptop. It keeps the always-needed dense weights resident, holds a 25% cache of experts in memory and reads the rest from the SSD on demand, with a prefetch predictor for the next layers.
The repository → 0 stars · Python
| Field | Value |
|---|---|
| Repository | antonio-cavalcante/mlx-moe-ssd-streaming 0 stars · read 2026-10-17 |
| Language | Python Last push 2026-10-17 |
| Licence | MIT Repository licence; the model weights carry their own. |
| Stack | Python on MLX and mlx-lm; mlx-community oQ3e checkpoint What it runs on. |
| Best for | Readers deciding whether an 816B model can run on hardware they already own, and what the tradeoff costs in tokens per second. A reader, not a benchmark. |
How it works
The layer sits on top of mlx-lm's own muse_spark model code rather than forking it. Decode looks up the routed experts for each token; about 83-84% of lookups hit the in-memory slabs, 5% are prefetched just in time and 11% miss to disk. The author measures the result with preadv counters cross-checked against iostat.
What we checked
- The README's results tables for decode, prompt processing and memory
- The explanation of why an 88% hit rate still means the SSD is busy
- The benchmark method: three greedy 250-token generations from 160-190-token prompts
- The file list for run_stream.py and bench.py
What we did not check
We did not run it, and the numbers are the author's single-machine measurements on an M5 Max with 128 GB, not a reproduction.
Our note
The most interesting experiment in the first 48 hours: keep 17 GB of dense weights in RAM, stream the routed experts from an SSD, and run an 816B model on a 128 GB MacBook at about 4.4 tokens per second.
github.com/antonio-cavalcante/mlx-moe-ssd-streaming Verified 2026-10-17 Stars 0 at read time