Launch HN: Magnitude (YC S25) – Self-optimizing inference engine for agents | AI Paper Digest

TL;DR Highlight

Open-source engine tunes kernels per model on your hardware, claiming 2x Ollama speed on average

Who Should Read

Developers running local LLMs behind coding agents who are unhappy with Ollama or llama.cpp speed

Core Mechanics

It tunes kernels per model on your chip
- On model download it spends up to a minute trying different ways to split up the math.
- It keeps the fastest combination that doesn't change the model's output.
Claims over 2x faster than Ollama
- The average is across popular local models on Mac and NVIDIA.
- Full benchmarks are on a separate page.
Ships a speculative decoding drafter per model
- Each model comes with the fastest drafter for speculative decoding.
Also handles memory and concurrency
- Quantized KV cache halves memory per agent.
- Sessions share prefix caches so concurrent use doesn't slow down.
Easy agent hookup
- One click connects Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline.
- Anything else works via the OpenAI-compatible API.

Community Reactions

Slower than rapid-mlx on an M5 Max
- A commenter averaged several 4-bit models: rapid-mlx decoded at 175 tok/s, Magnitude 0.2.1 at 161 tok/s.
- Their view is it adds little for already-optimized models.
Faster decode but slower prefill
- A commenter on an M5 Pro 48GB with gemma-4-26b-qat-4bit saw 82.8 vs 76.5 tok/s decode against oMLX.
- Prefill was about 2.6x slower, 709 vs 1843 tok/s.
Baseline criticism: llama.cpp and Ollama are low bars
- Several commenters say MLX should be the baseline on Mac.
- Others ask for benchmark methodology and results at 100-200K context.
Multi-GPU detection problems
- One commenter's two 16GB NVIDIA GPUs were detected as four, with only one used.
- They report llama.cpp decoding 20-30% faster on their 5070ti.

How to Apply

Compare speed against your current engine with the same model
- Install Magnitude today and download the model you already use.
- Run the same prompt on both and measure decode tok/s, TTFT, and prefill time.
Benchmark with long context
- Test at 100K+ tokens, the length your agents really use.
- Don't trust short-context results alone.
Point your existing agent at the OpenAI-compatible endpoint
- Change only the base URL in your runtime to Magnitude.
- Check that response quality and tool calling behave the same.

Try It

copy & run

# Paste into Claude Code or ChatGPT: ask for a benchmark script for local engines (simplified version of the article's idea)

Write a Python script that compares two local LLM engines (A: {{existing engine base URL}}, B: {{Magnitude base URL}}) through the OpenAI-compatible API.
- Model: {{model name}}
- Prompt lengths: 4K, 32K, 128K tokens
- Run each 3 times and print average TTFT, decode tok/s, and total time in a table
- Measure first-token time with streaming

Next step: also send the same system prompt twice in a row and measure whether the second TTFT drops thanks to prefix caching.

Terminology

KernelA small program that does the actual matrix math on the GPU. The same computation can differ a lot in speed depending on how threads are split.

Speculative decodingA small model drafts a few tokens and the big model verifies them in one pass. It is faster when the draft is right.

KV cacheMemory that stores computed results for context already read. It grows with context length.

Prefix cacheA cache that reuses computation for an identical prompt beginning. Useful for agents that resend the same system prompt.

TTFTTime to first token after a request is sent.

Launch HN: Magnitude (YC S25) – Self-optimizing inference engine for agents