Reviewer Capability Governs Rejection Targeting, Not Repair Skill: Evidence from LLM Execute-Review-Revise Pipelines
TL;DR Highlight
A cross-family mid-tier reviewer lifted accuracy from 52% to 64%, while same-model self-review gave no significant gain.
Who Should Read
Developers building execute-review-revise agent pipelines who mix cheap and strong models and must choose the reviewer model.
Core Mechanics
- A cross-family reviewer raised accuracy
- The Gemini 3.1 Flash-Lite executor alone scored 52%.
- With gpt-oss-20b as reviewer it reached 64% with zero damaged answers.
- Self-review detects a lot but gains little
- Same-model self-review had 0.85 recall but only +6pp accuracy, not significant.
- It rejected 2.1x as often with a 15% repair rate versus 43% for the cross-family reviewer.
- Self-review rejects correct answers often
- It rejected 35% (18 of 52) of correct answers.
- The cross-family reviewer rejected only 2% (1 of 52).
- Low damage came from the executor ignoring critiques
- Of 18 false rejections, the 3 the executor followed all turned wrong.
- The 15 it ignored stayed correct, so self-review only looked safe.
- Too-weak reviewers just burn tokens
Evidence
- Accuracy on 100 Omni-MATH problems
- No review 52%, self-review 58% (p=0.146), cross-family 64% (p=0.0005), weak 52%.
- Cross-family fixed 12 problems and broke 0.
- Rejection outcomes, self-review vs cross-family
- Rejections were 59 vs 28, with repair rates of 15% vs 43% (Fisher p=0.0074).
- No-change rates were 41% vs 14% (p=0.0151).
- Efficiency and cost
- Rejections per useful repair were 6.6 vs 2.3.
- Tokens per extra correct answer were about 25,600 (self) vs 23,700 (cross-family).
- Weak reviewer waste
- It made only 2 rejections with recall 0.04.
- Tokens per task rose from 668 to 1,353.
How to Apply
- Swap in a reviewer from a different model family
- If your pipeline uses self-review, replace only the reviewer with a mid-tier model from another vendor.
- Run the same executor outputs through both reviewers on 100 samples and compare accuracy.
- Log every rejection in four outcomes
- Record repaired, damaged, no change, or changed to a different wrong answer.
- Compute the false-rejection rate separately on a labeled eval set.
- Pick reviewers by precision, not recall
- Compare repairs per rejection across candidate reviewers.
- Drop models with many rejections and a high false-rejection rate.
- Measure change rate before adding a weak reviewer
- Enable the review step and check how often final answers actually change and how many tokens it adds.
- If answers barely change while cost rises, remove the step.
Try It
# Paste into Claude or ChatGPT (simplified version of the paper's method: a different model as reviewer)
You are a reviewer. Given the problem and solution below, reject only if the solution is wrong. If unsure, accept. You are not given the ground truth.
Problem: {{problem}}
Solution: {{solution}}
Output format:
VERDICT: accept or reject
CRITIQUE: if reject, name the wrong step and why; if accept, one line
Next: if it rejects, give the solution and CRITIQUE back to the original executor for one revision, then compare with the ground truth and count repaired / damaged / unchanged.Terminology
Related Papers
Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.
Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents
Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.
Harness Engineering 101: The LLM is the brain, the harness is the body
An agent is a loop managing a JSON array; 16 chapters build one from scratch
Harness2 by Google Research – Recursive Agent Harnessing
Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.
Ousted Savannah police officer gave ICE agent husband access to Flock system
A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired
Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling
Running Claude Code agents in tmux works well, but limits appear around four agents.
Related Resources
Original Abstract (Expand)
Multi-agent LLM pipelines increasingly assign roles, including execution and verification, to models of different capability tiers. This is done because running a flagship model at every stage is expensive. Previous literature has established that verification stages are not always beneficial, but holds reviewer capability roughly fixed relative to the executor. We vary it. We replace the reviewer with models spanning a capability range down to one that cannot solve the problems at all, and measure the outcome of every individual rejection. This is done across a constant set of 100 olympiad mathematics problems. A cross-family mid-tier reviewer improves final accuracy by 12 percentage points, from 52 to 64 percent (p = 0.0005), with zero damaged answers. Same-model self-review attains the highest error-detection rate of any condition (0.85 recall) yet yields no significant gain: it rejects 2.1 times as often for a third the repair rate (15 against 43 percent, p = 0.0074) and falsely rejects 35 percent of its own correct answers against 2 percent for the cross-family reviewer (paired p = 0.000015). The low damage rate of self-review proves to be an artifact of revision inertia rather than reviewer quality: of 18 falsely rejected correct answers, the three where the executor complied all became wrong, while the fifteen it ignored survived unchanged. Below a capability floor the role becomes inert: our weakest reviewer changed zero of 100 final answers while doubling token cost. These findings describe a single executor-reviewer configuration on 100 problems and should be read as a controlled pilot rather than a general claim about verification stages.