Janas: Run LLMs larger than RAM by streaming experts from SSD

TL;DR Highlight

Streaming MoE experts from SSD ran an 80B model on a 32GB laptop at 24 tok/s versus llama.cpp's 2.8.

Who Should Read

A developer who wants to run large MoE models locally on a 32GB-RAM laptop without a big GPU

Core Mechanics

Models bigger than RAM run by reading from SSD
- Janas-LLM keeps weights on disk and streams the experts each token needs from an NVMe SSD.
- Experts that keep coming back stay in a memory cache.
- This works because MoE (Mixture-of-Experts, where only some expert blocks run per token) activates only part of the model.
Qwen3-Next-80B beat llama.cpp
- On the same laptop (32GB, Core Ultra 9 185H, integrated Arc GPU), the 48.4GB model wrote at 24 tok/s versus llama.cpp's 2.8 tok/s.
- Reading a 1,024-token prompt was 62 versus 16 tok/s.
- With drafts (a small predictor guessing next tokens that the model verifies) writing reached 37 tok/s.
Faster on most of the 13 models compared
- The README says Janas is faster than llama.cpp on all 13 models in Q4_K_M.
- For example, Qwen3-30B-A3B wrote at 32 versus 15 tok/s and read at 135 versus 122 tok/s.
- The exception is 3-bit Qwen3-30B, which read prompts slower (100 versus 112 tok/s).
llama.cpp ran out of memory on the big model

Community Reactions

Opinion that this reinvents existing tools
- One commenter said people keep reimplementing what llama.cpp has had for years.
- Another said there are many SSD expert-streaming projects and a guide or merged project is needed.
Experience hoping for faster local inference
- A user who reached about 8 tok/s on Qwen 35B A3B with 32GB RAM said this could be a game changer.
- Another comment said 62 tok/s is enough for ordinary local use.
Experience with GPU OOM crashes
- A commenter said LLM-assisted development had crashed their display via OOM many times.
- They now reserve space for the display and check before allocating.
Concerns about SSD wear and model choice
- Some worried the heavy reads would wear out the SSD.
- Another asked why such old models were used instead of something like Qwen 3.8.

How to Apply

Measure your own machine with janas-bench
- On an idle Linux x86-64 box, download the release tarball and run janas-chat.
- Start with a small model and compare read/write tok/s from janas-bench against llama-bench.
- Set --cache <GiB> to cap memory from the start.
Set safety limits first
- On a test machine, use JANAS_GPU=0 and a --cache limit to reduce swap and driver-reset risk.
- Measure with other apps closed, then again with a browser open.
Gauge local candidates by SSD speed
- When a model file exceeds RAM, check your NVMe read speed first (the author's was about 5.7 GB/s).
- Compare it with the active expert size to judge whether local deployment is realistic.
Try it as a local code completion server
- Start janas-server and point Continue or llama.vscode at its OpenAI-compatible endpoint in VS Code.
- Time a single completion with a code model like Qwen3-Coder-30B-A3B.

Try It

copy & run

# Paste into a terminal (Linux x86-64). Steps are taken from the README

curl -LO https://github.com/prabanta-dev/janas/releases/latest/download/janas-linux-x86_64.tar.gz
tar xf janas-linux-x86_64.tar.gz && cd janas

# Run with a memory cap and pick a model (prefix JANAS_GPU=0 to keep it off the GPU)
./janas-chat --cache 16

# Next: run janas-bench to measure read/write speed on your machine and compare with llama-bench

Terminology

MoE (Mixture-of-Experts)A model with many expert blocks where only a few run per token. Like pulling only the shelves you need from a library.

Expert streamingReading only the needed experts from SSD instead of loading them all into RAM. Frequently used ones stay cached in memory.

Drafts (speculative decoding)A small predictor guesses several next tokens and the main model verifies them at once. Correct guesses give the same output faster.

O_DIRECTReads disk directly, bypassing the OS page cache. It is fast, but the system can slow down when memory headroom is small.

Q4_K_MA GGUF quantization format that shrinks weights to about 4 bits. It balances size against quality loss.