Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

TL;DR Highlight

A local engine runs a 125B model on a 32GB gaming PC with OpenAI/Anthropic-compatible APIs.

Who Should Read

Backend developers who want to run coding agents on a local GPU for cost and privacy reasons.

Core Mechanics

A 125B model on an ordinary PC
- Strata is an inference engine that runs the server-class Qwen3.8-Flash-Next (125B parameters) on a gaming PC.
- You need at least 32GB of RAM and VRAM combined.
- About 80GB of free disk is required.
Compatible APIs let existing tools connect
- It serves the OpenAI, Anthropic and OpenAI Responses APIs on localhost.
- Claude Code, Codex CLI and OpenCode can connect.
- For Claude Code, set ANTHROPIC_BASE_URL=http://127.0.0.1:8080.
Experts are spread over VRAM, RAM and SSD
- The model has 24,576 experts (small sub-networks in an MoE design) and each token uses 10.
- The roughly 4,000 most used sit in VRAM, all of them in RAM, and the rest are read from SSD.
- This fits the model into 12-24GB of VRAM.
Q2_0 hits 94 tokens/s on an RTX 5070 (12GB)
- Measured on an RTX 5070, Ryzen 5 7600 and 64GB RAM.

Community Reactions

Many doubt the quality of low-bit quants
- Some worried that quantizing below 4 bits degrades quality significantly.
- Others said Q2 is not interesting or not reliable for long-horizon coding.
- One suggested headlines should state the quantization.
A test found vision accuracy worse than llama.cpp
- On a 50-image object coordinate benchmark Strata had a median error of 154.8 pixels.
- The same GGUF and vision adapter on llama.cpp gave 46.5.
- The commenter said the gap is as big as going from a 9B to a 35B model.
Real-world speed and satisfaction varied
- One got 124 tokens/s on an RTX 4090 with 128GB DDR5.
- Another got about 60 t/s on an R9700 32GB with 96GB DDR4, about 2x faster than Qwen-3.8-27B.
- Another used it on a 3090 for PHP security audits and was happy.
Coding quality and concurrency disappointed some
- On a 4090 16GB with 96GB RAM, the Coder model failed a Rust binary tree task.

How to Apply

Install Strata and start a localhost server
- Today, on a PC with 32GB+ of RAM and VRAM, run install.sh or install.ps1.
- Start the server with strata launch --model q2_0 --yes --no-tui.
- Send a curl request to http://127.0.0.1:8080/v1 to confirm it responds.
Point your existing agent at the local URL
- For Claude Code, set ANTHROPIC_BASE_URL=http://127.0.0.1:8080.
- For other tools, run strata connect and apply the printed settings.
- Give it one small repo task and compare with your cloud result.
Compare quant sizes on your own workload
- Run the same 10 tasks on two sizes such as Q2_0 and IQ3_XXS.
- Record speed (tokens/s) and correctness in a table.
- If you plan to use vision, compare accuracy against llama.cpp separately.

Try It

copy & run

# Paste into a terminal (Linux/Mac) — install Strata and test the local server

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh

strata launch --model q2_0 --yes --no-tui

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"any","messages":[{"role":"user","content":"{{question}}"}]}'

# Next: watch speed and memory with strata monitor --url http://127.0.0.1:8080

Terminology

MoE (Mixture of Experts)An architecture where only some of many expert networks are used per token. Like opening 10 of 24,576 reference books per word.

QuantizationCompressing weights to 2-4 bits to save memory. Like compressing a photo: smaller file, lower fidelity.

VRAMDedicated memory on the GPU. The small but fast desk.

Prefill / decodePrefill reads the prompt and decode writes the answer token by token. The article benchmarks them as reading and answering speed.

GGUFA quantized model file format used in the llama.cpp ecosystem.