Why your local LLM feels dumber than it is

TL;DR Highlight

Changing only attention backend, KV cache, or quantization broke tool calls in local LLMs.

Who Should Read

A backend developer running a local model with vLLM for agent tool calls who is puzzled by intermittent failures

Core Mechanics

Same weights, different results
- Different GPU generations and instruction sets compute next-token math differently even with identical weights.
- The author notes the nightly vLLM container had 734 packages, 252 of them Python.
Swapping attention backend flips tokens
- Tested on Qwen3.6-27B BF16, RTX PRO 6000 Blackwell, a ~100k-token prompt, with Triton as baseline against FlashAttention 2 and FlashInfer.
- The first several thousand tokens matched, then top-1 tokens started to disagree in later portions.
- Repeated runs of the same backend gave bit-identical logits, so this is not random noise.
KV cache quantization breaks tool calls
- With BF16 weights and only the KV cache quantized, divergence appeared.
- int8 eventually recovered but int4 did not, giving a reproducible tool-call error.
4-bit quants failed tool calls more often
- In the five-way comparison, INT8 W8A16 had fewer token flips than official FP8 (W8A8) and NVIDIA NVFP4.
- NVFP4 came last, with about 50% token flips near 88k context.

Community Reactions

A llama.cpp parser bug example
- A commenter debugged a reasoning loop in Step 3.7 Flash on llama.cpp caused by the parser capturing an extra `\n` in the reasoning block.
- It reportedly only showed up in long multi-turn agentic sessions.
Opinion: don't quantize the KV cache
- One view was that if the checkpoint wasn't trained with quantized KV caches, you shouldn't quantize them at inference.
- Another commenter's rule is no KV quantization and nothing worse than the best Q8, preferring slower but more accurate.
Opinion: llama.cpp grammar enforcement avoids some failures
- A commenter said llama.cpp and ik_llama.cpp enforce grammar once a tool call is detected, so unclosed tool calls can't happen.
- Wrong command choices can still occur, they noted.
A question about cloud-hosted models
- A commenter asked whether this applies to cloud LLMs, citing day vs evening quality differences and 'dumb' resumed sessions, and wondered about KV cache eviction policies.
- No answer was given.

How to Apply

Turn off KV cache quantization first
- Set the KV cache dtype back to BF16 on the local server with intermittent tool-call failures today.
- Rerun at least 5 failing cases and compare.
Run a tool-call regression test on every change
- Collect real agent session logs as a fixed test set.
- Change attention backend, quantization, and tensor parallelism one at a time and compare tool-call success rates.
Match sampler settings to the model card
- Compare temp, top-p, and chat template on the Hugging Face model card against your server config.
- Check whether too-low temp causes THINK loops.
Check disclosures before trusting a quant's KLD claim
- Look for the reference checkpoint, eval text, context lengths, and KL direction.
- If undisclosed, don't pick a model on that number; evaluate on your own workload.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the post's method: a tool-call regression checklist)

I run a local model with vLLM and use agent tool calls. Tool calls sometimes fail in this setup.
- Model: {{model name}}
- Quantization: {{quantization}}
- KV cache dtype: {{KV dtype}}
- Attention backend: {{backend}}
- Tensor parallelism: {{TP value}}
- Failure log: {{failed tool call log}}

Make a table of experiments that change one variable at a time to narrow the cause. For each, include 'setting to change', 'settings to keep fixed', 'success criterion (tool call arguments match the correct ones)', and 'number of repeats'.

Next: run the experiment that sets the KV cache back to BF16 first, then paste the results and ask for the next step.

Terminology

KLD (KL Divergence)A measure of how far one probability distribution is from another. Lower means closer to the baseline, not smarter.

LogitsThe scores a model gives each candidate next token. They become probabilities, and one is turned into text.

Top-1 flipWhen the highest-scoring next token differs from the baseline run, like the top answer on a quiz changing.

KV cacheMemory that stores already-computed results for the prompt. It grows with length, so people quantize it to save space.

Attention backendThe implementation that actually computes attention (FlashAttention 2, FlashInfer, Triton). Same math, but rounding differs slightly per implementation.

Tensor parallelism (TP)Splitting one layer's computation across multiple GPUs. The order of operations when merging can change.