From the creator of Redis; run LLM locally with ds4

TL;DR Highlight

antirez's C engine ds4 runs DeepSeek V4, Qwen and GLM locally on Mac and CUDA

Who Should Read

A backend developer who wants to run coding agents on their own Mac or GPU box without API costs

Core Mechanics

ds4 is a model-specific C engine
- It is not a generic GGUF runner and validates only the supported model layouts end to end.
- It supports DeepSeek V4/V4.1 Flash, GLM 5.x and Qwen3.8 Flash Next.
Only the experts get 2-bit quantization
- To fit the 284B-parameter MoE DeepSeek V4 Flash locally, it uses asymmetric quantization.
- Only the routed experts are compressed while critical paths keep their precision.
KV cache is saved to disk
- Prompt prefixes are keyed by SHA1 and stored on SSD so a restart does not recompute them.
- This saves prefill time on long contexts.
One engine, three interfaces
- ./ds4 is a chat CLI, ./ds4-server a local API and ./ds4-agent a coding agent.
- The server exposes /v1/chat/completions, /v1/messages and /v1/responses so OpenCode, Claude Code and Codex CLI connect via a base URL.
Hardware starts at a 64GB Mac

Community Reactions

Context extension experiment
- One commenter wrote a fused TQ PR to fit 1M context on a 128GB M5 Max with Qwen 3.8 flash next.
- They also mentioned oMLX's Metal kernel speedups.
Daily use on M5 Max 128GB
- One user has run it since the DeepSeek V4 flash release and calls it the best launcher.
- They ran Qwen 3.8 flash next for over a week, fast with long context.
- The model sometimes forgets earlier statements, which might be the agent harness.
Pushback: how is it different from llama.cpp or ollama
- Several commenters asked what is new compared to ollama, which they already use locally.
- One commenter keeps a llama.cpp fork with disk KV cache.
Quality and memory requirement concerns
- One commenter says the heavily quantized dsv4 checkpoint is not very good.
- Another notes the blog says 64GB minimum while GitHub says 96GB.

How to Apply

Build ds4 and start a local API
- On a Mac with 64GB+, git clone, run ./download_model.sh ds4f-q2 and make.
- Start ./ds4-server --ctx 100000 and curl /v1/chat/completions to confirm a response.
Point your coding agent at the local base URL
- Change the base URL in Claude Code or OpenCode to the ds4-server address.
- Run the same repo task against a cloud model and compare speed and quality.
Use prefix caching for long system prompts
- Keep the system prompt and documents fixed at the start so the prefix hash matches.
- Measure first-response time before and after a server restart.
Pick a model that fits your machine
- Use the fit check on the Hardware page to choose platform and memory.
- Below 128GB, check first which models need SSD streaming.

Try It

copy & run

# Paste into a terminal (macOS, Apple Silicon 64GB+)

git clone https://github.com/antirez/ds4
cd ds4 && ./download_model.sh ds4f-q2
make

# Start the server
./ds4-server --ctx 100000

# Call from another terminal (put the port shown by the server in {{PORT}})
curl http://localhost:{{PORT}}/v1/chat/completions -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"{{QUESTION}}"}]}'

# Next: restart the server with the same long prompt and compare first-response time

Terminology

MoEMixture-of-Experts. Like calling in only a few specialists per task, so compute is small relative to total size.

QuantizationStoring weights in fewer bits to save memory, like lowering image resolution to shrink a file.

KV cacheSaved computation of the prompt already read. Like a bookmark so you don't reread from page one.

prefillThe stage where the whole prompt is read in. It is measured separately from token-by-token generation.

SSD streamingReading weights that don't fit in memory from SSD on demand.