Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures | AI Paper Digest

TL;DR Highlight

Prompting an LLM judge to keep searching unread evidence lifted Opus-4.8 F1 from 0.471 to 0.608.

Who Should Read

An agent developer whose execution logs have grown to hundreds of thousands of tokens and who wants LLM-based automatic failure root-cause analysis.

Core Mechanics

LLM judges settle too early
- On long logs the LLM locks onto a plausible cause and leaves the rest of the evidence unread.
- So the paper treats root-cause attribution as a search problem, not a one-shot judgment.
Continual Search keeps digging
- After the first verdict, later turns tell the judge to challenge its answer and read evidence it hasn't opened.
- It is compared against Passive Continuation, which only says to reconsider.
It only helps on long logs
- Scores rose on MegaRCA-Mix, TRAIL and TELBench, which have long logs.
- On the short AgentRx and Who&When it made results worse.
Resampling the same model doesn't help
- Self-consistency (majority vote over samples) scored below a single run on TRAIL.
- A four-model judge panel also fell short of Continual Search.
Search beats bigger models

Evidence

29% F1 gain on MegaRCA-Mix
- Opus-4.8 F1 went from 0.471 single-turn to 0.608 after four turns.
- Passive Continuation reached 0.479.
Method comparison on TRAIL
- Opus-4.8 Weighted F1: Continual Search 0.543, single-turn 0.482, self-consistency 0.430, judge panel 0.431.
- Cost was $3.43 for Continual Search versus $6.52 for self-consistency.
More evidence was actually read
- MegaRCA-Mix evidence coverage rose from 70.5% to 98.5%, while Passive plateaued at 75.6%.
- On TRAIL, deduplicated tokens read grew from 20K to 24K versus 21K for Passive.
Worse on short logs
- GPT-5.5 on AgentRx dropped from 0.310 to 0.241 and on Who&When from 0.516 to 0.437.
- Opus-4.8 on AgentRx dropped from 0.414 to 0.379.

How to Apply

Add follow-up turns only for long-log analysis
- When logs exceed tens of thousands of tokens, add 2-3 follow-up turns after the first verdict.
- Keep single-turn for logs of a few thousand tokens.
Give the judge agent file-reading tools
- Don't inline the whole log; give read-only tools so it opens files on demand.
- Pass the list of still-unopened files each turn and make it open all of them.
Require a quote to change the cause
- Let the judge change its answer only to an earlier cause backed by an exact quote.
- Without a quote, it must keep the current answer to resist pressure.
Spend the budget on turns, not resampling
- Instead of majority-voting several runs, spend the same budget on sequential search turns.
- Also compare giving a smaller model more turns against a bigger model.

Try It

copy & run

# Paste into Claude Code or Claude (simplified version of the paper's method: an agent environment with file access is recommended)

[Turn 1] Read the failed agent execution record in {{log folder}} and find the first point where it diverged from a correct run (the root cause). Answer with a label and a quote. The final error is only a symptom.

[Turn 2] That answer came from a single reading and is untested. Re-read from the start, derive the cause independently again, then keep whichever of the two conclusions has better evidence.

[Turns 3-4] Unopened files: {{unread file list}}. Open all of them in full. Unopened evidence cannot support a conclusion. Change your current answer only to an earlier cause supported by an exact quote. Without a quote, keep it unchanged.

Next to try: run the same log with a prompt that only says "reconsider and reverify" and compare the results.

Terminology

RCA (root-cause attribution)Finding where an agent run first went wrong. It's like finding the first bad commit in a bug hunt.

Passive ContinuationA baseline that just says "reconsider and reverify your answer" each turn. It doesn't ask the judge to read new evidence.

Self-consistencySampling several independent answers from the same model and taking a majority vote.

Agent-as-a-JudgeAn LLM evaluator that uses tools to open and inspect evidence before giving a verdict.

MegaRCA-MixThe paper's benchmark of 50 human-labeled failure trials with a median log size of 286K tokens.

Related Papers

Original Abstract (Expand)

The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29\%, from $0.471$ to $0.608$. More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.