Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
TL;DR Highlight
Simple LLM harnesses beat hand-crafted agents at microservice root cause analysis
Who Should Read
SRE or backend developers building LLM agents for microservice root cause analysis who are weighing complex SOP or role-agent designs
Core Mechanics
- Simple harnesses came out on top
- A one-call evidence package, a SQL agent and Codex reached PR@1 of 65–77% on three mid-tier models.
- Five published LLM RCA agents got 16–53% and non-LLM methods 6–39%.
- Fixed procedures skip decisive evidence
- FoA filled the fault description as "microservice incident", picked a generic SOP that never reads logs, and blamed cart.
- mABC walked the call chain and never compared inventory, which had stopped receiving calls.
- Prescribed model calls mark the divide
- The three simple harnesses had zero program-decided model calls.
- Published agents ranged from at least 0.10 for mABC, 0.29 RCA-Agent, 0.35 RCAgent, 0.51 FoA to 1.00 DecompRCA.
- Stronger model buys reliability, multi-turn buys aptitude
- From Luna to Sol, A (solved at least once) rose only 2–7 points while U (solved only sometimes) fell from 32–36 to 7–8.
- On Luna, the SQL agent and Codex solved 10–11 points more cases at least once than one-shot.
- Reasoning effort mostly adds time
- Raising effort above medium multiplied wall time by 1.5 to 7 and changed RR by −2 to +7 points.
- Swapping Luna for Sol at medium raised RR by 13 to 15 points.
- Every cost/time Pareto-optimal run was a simple harness
- Among runs more accurate than NSigma (38.9%), all Pareto-optimal runs in cost or time used a simple harness.
- DecompRCA was cheapest and fastest but fell below the threshold at 24–29%.
Evidence
- Mean PR@1 over three mid-tier models
- Codex 77.2%, SQL agent 72.8%, one-shot 65.3%.
- Best published agent FoA 53.3%, best non-LLM NSigma 38.9%, random ranking 5.5% (RCAtlas, 90 cases).
- Even one-shot beat FoA
- The weakest simple harness, one-shot, led the best published agent FoA by 12 points.
- Across six models, simple harnesses scored 60–71% on Luna and Haiku and 74–87% on Flash, Sol and Astra.
- Cost and speed examples
- Luna with Codex reached 68.9% for under one cent per case, and Flash with Codex 85.6% for about five cents.
- Sol with the SQL agent hit 85.6% in 45 s, and Sol xhigh with Codex hit 88.9% at about 26 cents and 277 s per case.
- Stronger model versus more effort
- Luna to Sol raised PR@1 by 14–18 points.
- Luna medium to max, and Sol medium to xhigh, gained 2–4 points at 1.5 to 4 times the wall time.
How to Apply
- Feed incident data to an LLM in one call
- Today, take one past incident and pack changed metrics, error log templates and call statistics into a single prompt asking for at most five root-cause candidates.
- Check it first on an incident whose answer you know.
- Turn fixed SOP steps into optional guidance
- If your agent only runs prescribed procedures, move the procedure docs into tool descriptions or reference notes so the model decides whether to read them.
- First check whether any step skips logs or traces.
- Upgrade the model, keep effort at medium
- If accuracy falls short, run the same harness on a one-tier-larger model before raising effort.
- Record time and cost together.
- Compare methods over three runs
- Weaker models are noisy, so run each case three times and look at the mean and the worst pass.
- Do not conclude from a single run.
Try It
# Paste into Claude or ChatGPT as is (simplified version of the paper's one-shot evidence package method)
You are a microservice incident root cause analyst. Using only the evidence below, rank up to 5 root-cause entities.
[Incident window] {{start and end time}}
[Services and call relations] {{inventory}}
[Metrics that changed a lot] {{metric changes}}
[Per-minute trace stats] {{call and error stats}}
[Log templates (count before/after)] {{logs}}
[Kubernetes events] {{events}}
Rules: do not assume the service with the most errors is the cause. Also check services whose calls stopped or whose request count collapsed. Cite the evidence items you used.
Next step: run the same evidence three times and check whether the top-1 answer is consistent.Terminology
Related Papers
GPT-5.6 Sol can beat Factorio on peaceful mode
GPT-5.6 Sol in Codex at low thinking finished a peaceful Factorio run with a rocket launch
AI isn't replacing SREs, it's deskilling them
Letting AI handle 95% of incidents may dull the skills SREs need for the critical 5%.
Backdoor in Claude Code for VS Code extension on the Cursor Marketplace
A tampered Claude Code extension on Cursor Marketplace planted an SSH backdoor
Building an AI-native engineering org
Automating each shipping step raised merges per engineer about 5x with reverts under 0.5%
We ran AI agent swarms in Minecraft and Mindustry
We put 50 agents in Minecraft and they invented ration ledgers and bureaucracy.
The Agent SDK
Cursor released Agent SDK, letting you define agents in code and deploy them with one command.
Here be monsters: What do you still need to know when agents do the work?
Owning agent-written code depends on fundamentals, breadth, and diving deep just in time.
AI promised less work. I've never worked more
Running agents in parallel brought my GitHub activity back to full-time engineer levels, and I'm exhausted.
Related Resources
Original Abstract (Expand)
Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a capable model investigate on its own. We compare three simple harnesses that prescribe none of the model's calls (one call over an evidence package, a SQL agent and the Codex coding agent) with five published LLM RCA agents and with statistical and causal methods on RCAtlas, 90 validated cases from eight public datasets, across six models and three reasoning efforts. The simple harnesses are the most accurate (PR@1 65-77% on three mid-tier models), ahead of the published agents (16-53%) and the non-LLM methods (6-39%), on every model on which both families were run. The published agents fix from a tenth to all of their model calls in the program, and a case study shows how these prescriptions skip the decisive evidence. A stronger model mainly improves reliability, a multi-turn harness mainly improves aptitude, and more reasoning effort changes accuracy by a few points at most while multiplying the time by up to seven. Among runs more accurate than the best non-LLM method, every Pareto-optimal run in cost or time uses a simple harness.