Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

Who Should Read

Infra or backend developers who want to run large MoE models on local CPU servers or PCs without a GPU

Core Mechanics

• 284B model shrunk to 56.56 GiB
  • DeepSeek-V4-Flash is an MoE (only some experts run per token) with 284.33B total and 13.17B active parameters.
  • The checkpoint drops from 76.87 GiB (Unsloth UD-IQ1_S) to 56.56 GiB at 1.71 bpw.
  • Peak VmRSS (resident memory) at 256K context is 57.61 GiB.
• Fits in 64GB RAM with no swap
  • The original UD-IQ1_S exceeds the ~62.5 GiB usable on a 64GB Linux box and causes swap or OOM kills, per the README.
  • This checkpoint loads fully into RAM with zero swap.
  • By default all 6 routed experts per layer (Top-6) are evaluated with no pruning.
• CPU kernel patches gave +77% prefill
  • Prefill on a 16-core Zen 5 went from 25.0 to 44.36 tok/s.
  • The patches tile the matrix math across registers, L1, L2 and L3.
  • Decode gain is listed as +53% for the same patch set.
• Generation runs at 9 to 15 tok/s
  • Default Top-6 decode is 9.06 to 9.95 tok/s.
  • With Stockfish-PVS speculative decoding (a small drafter guesses ahead) it reaches 11.5 to 14.7 tok/s.
  • Top-4 and similar pruning options trade a bit of perplexity for speed.
• 30B MoE is practical on ordinary CPUs
  • Nemotron-3.5-30B-A3B Fractal-64 (14.35 GiB) runs at 25.0 to 60.4 tok/s in 16 to 32 GB RAM.
  • A Threadripper 7985WX reached 111.62 tok/s on the 30B.
  • A Windows GPU+CPU hybrid mode also exists.
• KV restore for Claude Code takes 13 ms
  • KV cache slots are snapshotted to /dev/shm and restored from there.
  • A 40-tool Claude Code prompt restores in 13.1 to 32.6 ms, per the README.
  • Custom tensor types mean stock llama.cpp, Ollama and LM Studio cannot load these files.

Community Reactions

• Experience that it matched plain llama.cpp
  • A commenter on a Ryzen 7945HX with 96GB ran Nemotron and found decode about the same as stock llama.cpp, with only prefill faster.
  • They could not find a DeepSeek option in setup, so they did not test it.
• Skepticism about output quality
  • A commenter who used DeepSeek V4 Flash heavily argued quality at this size is doubtful and likely worse than an Unsloth Q2XS quant.
  • Another cited research saying performance drops sharply below 4-bit quantization.
• Experience that it was usable on low-end hardware
  • A commenter on a Ryzen 7 5700U with 64GB DDR4 reported a steady 12 to 14 tok/s at 15W.
  • They said that is enough for personal work.
• Complaints about README readability
  • Several commenters said the AI-style README with many emojis is hard to read.
  • Others complained that the quantization bit width is hard to find.

How to Apply

• Check CPU readiness first
  • Run the readiness script to inspect your CPU and HugePages without downloading a model.
  • Execute the command in the codeExample on Linux.
• Serve the 30B model first
  • Download the Nemotron Fractal-64 GGUF and launch llama-server-modchip on a 16 to 32 GB RAM machine.
  • Point your existing OpenAI-compatible calls at port 8080 and measure latency.
• Run the quality gate on your own prompts
  • Run ./modchip eval for the 100-point quality and tool-calling gate.
  • Compare accuracy on your own prompt set before deciding on the 284B model.
• Measure agent context restore time
  • Run ./modchip claude and time the warm restore of the 40-tool prompt.
  • Compare it with first-response latency of your agent's long system prompt.

Try It

# Paste into a Linux terminal (public commands from the README, unchanged)

curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3

# Serve the 30B Nemotron (16-32GB RAM)
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF llama-server-modchip-linux-x86_64 NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf --local-dir .
chmod +x llama-server-modchip-linux-x86_64
./llama-server-modchip-linux-x86_64 -m NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf -c 262144 -t 16 --port 8080

# Next: auto setup and benchmark
git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip bench

Terminology

Related Papers

• Gemini Omni 1.2 Flash

  Gemini Omni 1.2 Flash makes 15-second videos with chat editing at 30% lower price than 1.1

• Progress on provable inference for large language models

  Proving 400B model outputs with zkSNARKs works, at 70-95x the GPU time of plain vLLM

• Tell HN: Gemini trains on chat history, even for subscribers (personal accounts)

  Gemini personal accounts may train on and human-review chats, even for paying subscribers

• You don't need long context

  Compacting context to 16K–48K kept accuracy while cutting estimated costs 2–8x

• The AI cost of JavaScript frameworks

  Building the same app with 12 JS frameworks via agents, 5 runs each, showed up to ~2x cost differences.

• Claude Code regularly sends each background agent’s transcript to its model

  Disabling Claude Code's 30-second agent status summaries removed 19.7% of API requests in one team's telemetry

• Cross-request KV cache exposure in the GLM-5.3 Workers AI model (155 responses)

  A KV pool smaller than the context window leaked other customers' data into 155 long-prompt responses.

• Microsoft Decision-1: 9B Qwen Model Launch Explained

  Microsoft's 9B Qwen-based model returns only per-option probabilities for routing and classification.

Related Resources

• https://github.com/synchopate/llama-modchip
• https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
• https://arxiv.org/pdf/2607.10855
• https://x.com/antirez/status/2119392658661749175