Asymmetric Capacity Allocation in Self-Refinement Pipelines

Who Should Read

An LLM app developer running a generate-critique-revise agent loop who wants to cut inference cost.

Core Mechanics

• Bigger generator and refiner help
  • Scaling either stage raised pipeline performance on all 5 benchmarks.
  • Their performance spread was much larger than the critic's.
• Critic size barely matters
  • The critic (the step that points out flaws in the draft) showed the lowest variability across all benchmarks and both model families.
  • A 32B critic found more errors, but final gains were small.
• Still, don't drop the critic
  • Even the smallest Qwen3-0.6B critic beat a pipeline where the refiner just revises with no critique.
  • The benefit comes from the critique content, not just an extra call.
• Small refiners wreck drafts
  • Under Qwen3-32B, 12 of 30 refiner pipelines scored below the initial generation.
  • The 0.6B refiner altered correct parts or followed wrong critiques.
• Big refiners edit only what's needed

Evidence

• Refiner variance exceeds critic's
  • On Qwen3 PIE, std is 20.60 for the refiner vs 1.63 for the critic.
  • Meeting Planning is 11.20 vs 3.10 and CNN/DailyMail is 8.99 vs 3.10.
• Gain from a 0.6B critic
  • Versus the no-critique baseline, Meeting Planning averages +9.67 (generator sweep) and +10.34 (refiner sweep).
  • PIE is +9.03/+11.06 and ZebraLogic is +7.97/+7.86.
• Degradation by small refiners
  • Pipelines below initial generation: Qwen3-32B 12/30, Qwen3-14B 9/30, Qwen3-8B 8/30, Gemma3-27B 8/20.
  • Of 50 degradation cases in P(32B,32B,0.6B), 41 happened despite non-misleading critiques (human score 3-5).
• Critique quality comparison
  • On ZebraLogic, non-misleading critiques were 70% for the 32B critic and 64% for the 0.6B critic.
  • 50 critiques per model were manually scored.

How to Apply

• Use a big refiner and a small critic
  • Swap your critique step to the smallest model and compare final scores on 20-50 identical inputs.
  • If the gap is small, keep the savings.
• Never put a small model in the revise step
  • If your refiner is small, move it to a model at least as large as the generator.
  • Measure how often the revision is worse than the draft.
• Add a minimal-edit rule to the refiner prompt
  • Tell it to fix only the flagged errors and leave the rest unchanged.
  • Also tell it to ignore critiques that look wrong.
• Run a critique on/off A/B test
  • Run the same refiner with and without critique.
  • Keep the small critic if the critique version wins.

Try It

# Paste into Claude/ChatGPT. Simplified version of the paper's method: refiner prompt (minimal-edit rule)

You are the refiner. Given the input, a draft, and a critique, produce a revised version.
Rule: fix only the errors flagged in the critique, and do not change anything correct. If the critique looks wrong, do not follow it and say why in one line.

[Input]
{{problem}}

[Draft]
{{draft}}

[Critique]
{{critique}}

Output the revision and a list of what you changed.

Next: feed the same draft to a small and a large refiner model and compare whether the draft gets broken.

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware