Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
TL;DR Highlight
A Rust-based open-source project that intelligently distributes LLM models across GPU, RAM, and NVMe when they exceed your Mac's physical memory, enabling models that crash llama.cpp with OOM errors to actually run.
Who Should Read
Developers with Apple Silicon Macs (MacBook Pro, Mac Studio, Mac Mini, etc.) who gave up running local LLMs due to memory constraints — especially ML engineers and AI researchers who want to experiment with 70B+ models on 32GB or less.
Core Mechanics
- The project implements smart tiered memory placement: model layers that are frequently accessed stay in GPU unified memory, less-used layers fall back to system RAM, and rarely-used layers spill to NVMe SSD.
- The tiering is dynamic — as different parts of the model are activated during inference, layers migrate between tiers based on access frequency and available memory.
- In benchmarks on a 32GB M3 Max, the project runs a 70B quantized model at 3-5 tokens/second — slow but functional, where llama.cpp fails entirely due to OOM.
- NVMe bandwidth on Apple Silicon (especially M-series Pro/Max/Ultra) is fast enough to make SSD spillover practical — the bandwidth isn't comparable to RAM but is sufficient for less-active layers.
- The project is in early stages and lacks some llama.cpp ecosystem integrations (certain quantization formats, sampling methods), but the core functionality works.
Evidence
- Benchmark videos showing 70B models running on 32GB MacBook Pros generated significant excitement — many developers had assumed this was simply impossible.
- Commenters with NVMe bandwidth knowledge validated the technical approach: Apple Silicon's NVMe is fast enough that SSD spillover is viable in ways it wouldn't be on typical PC SSDs.
- Some skepticism about real-world usefulness: 3-5 tokens/second is too slow for interactive use but might work for batch processing or offline generation tasks.
- Rust implementation was specifically called out as a smart choice for this use case — the memory management precision and performance characteristics align well with the problem.
How to Apply
- If you're on an Apple Silicon Mac with 32GB or less and want to run 70B models, this is currently the best option — try it for offline batch generation tasks where speed is less critical.
- Start with a Q4 quantized model to minimize the memory footprint — the tiering benefits are largest when the model fits mostly in RAM with only a small SSD overflow.
- Use it for experimentation and evaluation, not production serving — the current performance characteristics make it suitable for 'does this model behave the way I want?' testing.
- Monitor NVMe write cycles when using SSD spillover extensively — inference with heavy SSD use will wear down the drive faster than typical usage.
Terminology
Unified MemoryApple Silicon's architecture where CPU and GPU share the same physical memory pool, unlike discrete GPU systems with separate VRAM.
NVMe (Non-Volatile Memory Express)A high-speed SSD interface protocol — Apple Silicon Macs have some of the fastest NVMe implementations in consumer hardware.
OOM (Out of Memory)An error that occurs when a process requests more memory than is available, causing a crash.
QuantizationA technique that reduces model size by representing weights in lower precision (e.g., 4-bit instead of 16-bit), trading some accuracy for memory savings.
Tiered MemoryA memory hierarchy that places frequently-accessed data in fast storage (GPU memory) and less-accessed data in slower but larger storage (RAM, SSD).