llama.cpp
TL;DR Highlight
A 1MB app installs local models and exposes an OpenAI-compatible API in one click
Who Should Read
A backend developer who wants to run existing OpenAI API code on local models with no API keys or usage bills
Core Mechanics
- Llama is a local model runner app
- Built by the llama.cpp team and Hugging Face for Mac and Windows.
- Free, open source, works offline, and is a 1 MB download.
- Linux users get a command-line version via a curl install script.
- It exposes an OpenAI-compatible local API
- It listens at localhost:9931/v1, and changing the base URL is said to be enough for existing code.
- It offers OpenAI- and Anthropic-compatible endpoints.
- Streaming, tool calling, structured output, and vision are supported.
- Settings are picked automatically
- Quantization (compressing model weights) uses the highest precision that fits in memory.
- Context size, image input, and speculative decoding (using cheap guesses to speed up generation) are also automatic.
- Batch size is larger on Macs with 32 GB or more.
- Every llama.cpp setting can still be edited in one plain-text file.
- Each model is stored once and shared
Community Reactions
- llama-server already supports multiple models
- One commenter writes an ini file with per-model tuned parameters so clients just pick a model.
- They also suggest enabling ngram-mod (spec-default), calling it nearly free.
- Install tips for llama.app
- A commenter says it used to be called LlamaBarn.
- They suggest installing llama.cpp with brew first, since llama.app releases more slowly.
- Models downloaded with the hf CLI are picked up automatically.
- Stability complaints about llama.cpp
- One user on a Framework 13 says a code change broke ROCm support for the AMD GPU and stayed unfixed for nearly a month.
- They worked around it by switching to Vulkan.
- Setup experiences
- One user found llama.cpp easier to install than vLLM on an RTX 3070.
- Some are wary of curl | sh and prefer building from source.
- One user's Qwen3-4B Q4_0 server terminated while Llama-3.2-1B worked.
How to Apply
- Swap the base URL in your OpenAI code
- Install Llama today and download one model.
- Change the OpenAI SDK base URL to http://localhost:9931/v1 and use any placeholder API key.
- Check that your existing prompts return responses.
- Route only dev and test traffic to the local API
- Move the base URL into an environment variable.
- Send costly repeated test runs to a local model.
- First confirm tool calling and structured output work with your chosen model.
- Pick a model for your memory and compare quality
- Download one recommended model (Gemma 4, GPT-OSS, Qwen 3.8) that fits your Mac.
- Run 10 real prompts from your service on both your current API and the local model.
- Record the differences in a table.
Try It
# Paste into a terminal (after installing Llama and a model)
curl http://localhost:9931/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "ggml-org/gpt-oss-20b-GGUF:MXFP4",
"messages": [{"role": "user", "content": "{{question}}"}]
}'
# Next: point your existing OpenAI SDK code at http://localhost:9931/v1 by changing only the base URL.Terminology
Related Papers
Mercury 3: Diffusion at scale
Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality
Timestral: Mistral's time-series foundation models
A 209M time-series model ranks first on three benchmarks and runs on CPU.
DeepSeek V4.1 Flash 1017 (Max)
DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.
If you don't run your own LLM proxy, how do you know whats going on?
Running your own LLM proxy shows which keys, models, and cache misses drive your usage.
DeepSeek: DeepSeek v4.1 Flash 1017
DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents
DeepSeek-v4.1-Flash-1017
DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length
Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware