Apple Silicon and macOS VMs: Faster LLM Inference with llama.cpp

TL;DR Highlight

Changing Metal capability answers inside a macOS VM made llama.cpp up to 16x faster

Who Should Read

A developer running local LLMs with llama.cpp inside a macOS VM (e.g. Lume) on Apple Silicon who is frustrated by slow speeds

Core Mechanics

The VM under-reported GPU capability
- The macOS guest's virtual GPU reported roughly Apple family 5, 32 KB threadgroup memory, and no SIMD-group matrix support.
- llama.cpp read those answers and picked slower kernels.
A per-process shim changes only the answers
- The shim (a compatibility layer between app and API) reports Apple family 9 and 64 KB threadgroup memory.
- That turns on SIMD-group reduction, matrix, and bfloat16 paths.
TinyLlama 1.1B ran 11-16x faster
- On an M1 Ultra, prompt processing was 11.08x and token generation 16.36x versus the same stock VM.
- That is 98.25% (prompt) and 72.06% (generation) of bare-metal speed.
Gemma 4 12B and a 30B model also gained
- Gemma 4 12B QAT Q4_0 gained 7.20x prompt and 14.54x generation versus stock VM, reaching 99.59% and 94.82% of bare metal.
- Muse Glimmer 30B Q4_K-M in a 64 GiB guest gained 7.55x prompt and 8.87x generation versus stock VM.

Community Reactions

Metal 3 reporting broke MLX
- According to the post, advertising MTLGPUFamilyMetal3 during ablation made MLX request a residency set the virtual GPU lacks.
- So the release shim changes only Apple-family answers.
Opinion that the title reads like a general speedup
- Commenters noted this fixes kernel selection inside Virtualization.framework VMs, not llama.cpp on every Apple Silicon Mac.
- Some found the stock-VM baseline confusing in the title.
Question about why the VM reports a lower profile
- A commenter asked why Virtualization.framework does not report the host GPU's full capabilities, which the post does not explain.
Requests for other chips and bigger models
- Commenters asked about M1 Pro or M3 Pro results.
- One felt TinyLlama 1.1B alone is weak evidence.

How to Apply

Run the capability probe in your VM first
- Today, if you have a Lume VM, run build.sh and verify.sh in libs/lume/metal-capability-shim.
- Check that the stock guest reports Apple family 5.
Benchmark stock vs unlocked with the same model
- Run llama-bench with -p 512 -n 128 -r 10 in both setups and compare medians.
- Also measure on the host to get the ratio versus bare metal.
Inject the shim only into the inference server via LaunchAgent
- Set DYLD_INSERT_LIBRARIES in that workload's environment only.
- Keep the login session stock; to roll back, remove the variables and restart.
Re-validate after every macOS update
- Rerun the probe and benchmark whenever the host or guest version changes.
- Share chip, versions, and workload results in an issue.

Try It

copy & run

# Shell (host Mac). Steps from the post; replace my-vm and paths

cd libs/lume/metal-capability-shim
./Scripts/build.sh
./Scripts/verify.sh

lume stop my-vm
defaults write com.apple.gpusw.ParavirtualizedGraphics ForceUnrestrictedDeviceFeatureLevel -bool true
lume run my-vm

# After copying the dylib and probe into the guest, run
lume ssh my-vm "DYLD_INSERT_LIBRARIES=/path/to/LumeMetalCapabilities-arm64.dylib LUME_METAL_APPLE_FAMILY_MAX=1009 /path/to/metal-capabilities 1009"

# Next: benchmark with the same variables
lume ssh my-vm "DYLD_INSERT_LIBRARIES=/path/to/LumeMetalCapabilities-arm64.dylib LUME_METAL_APPLE_FAMILY_MAX=1009 llama-bench -m tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf -p 512 -n 128 -r 10 -t 8 -ngl -1 -o json"

Terminology

paravirtualizationThe guest uses a virtualization-aware device instead of emulated hardware, while the host keeps control of the real hardware.

GPU passthroughHanding a physical GPU wholesale to a VM. This post's method is not that; it uses the existing virtual GPU path.

Metal capability shimA thin layer that rewrites the answers when an app asks what the GPU supports.

threadgroup memoryA small shared memory for a group of GPU threads. Its limit influences which kernel gets chosen.

SIMD-group matrixA GPU hardware feature that speeds up matrix multiplication; when on, llama.cpp uses faster kernels.