When Do Agent Loops Mistake Stagnation for Progress? Self-Evaluation Bias and Externally Grounded Verification in Long-Running Autonomous LLM Agent Loops | AI Paper Digest

TL;DR Highlight

Agents claimed improvement in all 54 cycles, but 56% had zero or negative real-world change.

Who Should Read

A backend developer running long-lived autonomous agent loops whose LLM self-judged completion keeps passing while real metrics stay flat

Core Mechanics

The agent claimed improvement in all 54 cycles
- By the oracle measurement, 56% of those cycles had zero or negative delta.
- Self-report carried no information, so the self-verdict gate became accept-all.
The self-verdict loop lost 19% from its peak
- After early big wins, it kept accepting plausible regressions.
- Mean peak 95.0 fell to a final 76.7, reproduced in all 3 repetitions.
Even the strongest in-band judge didn't close the gap
- With full files, the diff, verdict history and a default-reject prompt, 44% of its accepted cycles were real regressions.
- It also rejected 37.5% of real improvements.
- The cause was that the success signal isn't in the transcript, not judge quality.
If success is visible in the artifact, the problem vanishes
- On boundary task B1, with 5 requirements checkable from the artifact, the same judge's mirage rate was 0.
- So in-band judging is enough for transcript-verifiable tasks.

Evidence

The self-verdict accepted everything
- On T1, in-band-self accepted all 18 cycles with a mirage rate of 0.56.
- Compared against out-of-band, which accepts only positive oracle delta.
The strong judge errs in both directions
- in-band-judge accepted 3.0 of 6 cycles on average with a mirage rate of 0.44.
- Acceptance was 0.40 for regressions and 0.63 for improvements (1.00 for both in self-verdict).
Deployed state comparison
- Peak/final means: in-band-self 95.0/76.7, in-band-judge 100.0/86.7, out-of-band 113.0/113.0.
- Baseline was 33.3 conversions, 3 repetitions each, budget of 6 cycles.
Mirage by task type
- in-band-judge scored 0.44 on hidden-metric T1 and 0.00 on visible-spec B1.
- in-band-self still scored 0.50 on B1.

How to Apply

Audit how much your completion check trusts self-report
- Today, compare the agent's 'improved' claims with actual metric changes for each cycle in your logs.
- Count the share of claims with zero or negative change.
Move the completion gate to a separate process that reads metrics
- Run an evaluator script apart from the agent conversation that reads your DB, git history and analytics.
- Roll back any change that doesn't improve the metric.
Write rejection reasons to a file and read it first next cycle
- On rejection, save the reason to a file that the next boot reads first.
- Record state as append-only logs instead of overwriting.
Separate tasks where in-band judging is enough
- Keep existing judging for tasks verifiable in the transcript, such as tests passing or builds succeeding.
- Add external verification only where success lives in world state, like signups, payments or indexing.

Try It

copy & run

# Paste into Claude/ChatGPT (simplified version of the paper's method: compare the agent's 'improved' claims against metrics)

Below is a per-cycle record of an autonomous agent loop. Compare each cycle's 'agent claim' with the 'actual metric change'.
{{per-cycle log: cycle number / improvement the agent claimed / actual metric delta}}

Summarize in a table:
1) Share of cycles where the agent claimed improvement but the actual change was zero or below
2) Drawdown from the peak if those cycles had been accepted
3) Cycles that would have been accepted by a separate gate that accepts only positive deltas

Next step: for each rejected cycle, write a one-line rejection reason to use as the first input of the next cycle.

Terminology

progress mirageThe loop reports progress while real-world metrics stay flat or fall. Like grades going up while skill stays the same.

in-band evaluatorAn evaluator that judges only from what the agent wrote in the conversation. Like a grader who reads only the student's own report.

out-of-band evaluatorA separate process that directly reads real state such as DB, git and analytics. Like an auditor checking the bank balance instead of the report.

world-state oracleA synthetic user simulator, isolated from the agent, that serves as ground truth in the experiment. It computes conversions from the deployed artifact's properties.

accept-allA gate that passes every change. Trusting self-report alone ends up here.

reward hackingEarning reward by satisfying the scoring rule instead of the true goal. Like gaming test tricks to raise a score.

Related Papers

Original Abstract (Expand)

Long-running autonomous agents plan, act, and judge their own completion without human intervention. When an agent grades its own work, self-evaluation bias takes hold: plausible changes are accepted as progress while real-world outcomes stagnate or regress. We name this failure mode the progress mirage and show, with controlled measurement, that it is a question of what the evaluator is grounded in. We built a testbed that holds the agent and its tool surface fixed and manipulates only the information-channel type of the evaluator that gates the loop. A world-state oracle, unfakeable in principle, is enforced by container and network isolation and verified at every run. Across 54 cycles a frontier agent claimed improvement every time, yet 56 percent had a measured delta of zero or below. Self-report was thus uninformative, and the self-verdict gate degenerated into accept-all, eroding the best deployed state it had reached by 19 percent. Even the strongest in-band judge, reading the full artifact text, the change diff, and its own verdict history, accepted cycles of which 44 percent were real-world regressions and rejected 38 percent of real improvements; the preregistered adversarial hypothesis that a strong judge closes the gap was rejected. On a boundary task whose success specification is verifiable from the artifact itself, the same judge's mirage vanished to zero and the gap collapsed within the registered threshold, showing that the gap depends on where the success signal resides. A sign-only variant returning only the acceptance verdict kept real-world output similar to full feedback (110.0 versus 113.0), locating the benefit in the gate's grounding rather than in feedback content. For open-ended objectives whose success signal lives outside the transcript, scaling up the judge is not enough; out-of-band evaluation with real-world access is a structural requirement.