What Confidence Routing Is Actually Doing: Auditing Routing, Calibration, and Commitment in Multi-Agent Deliberation

Who Should Read

Backend developers running multi-agent systems that use agents' confidence scores to pick the next speaker or answer

Core Mechanics

• One confidence score is doing three jobs
  • Stated confidence picks the speaker, estimates probability, and decides the public answer.
  • The paper says routing, calibration, and commitment must be measured separately.
• Confidence ranks well but is a bad probability
  • On gpt-oss-120b math traces, AUROC (ranking accuracy, 0.5 is chance) is 0.721.
  • Mean stated confidence is 0.790 versus 0.518 actual accuracy, so it is badly overconfident.
• Isotonic calibration fixes the scale cheaply
  • ECE (calibration gap) drops from 0.278 to 0.008 after calibration.
  • AUROC stays 0.721 to 0.720, so it cannot create missing discrimination.
• Swapping routing rules barely matters
  • Five fixed routers on gpt-oss/math are within 1.1 pp of each other.
  • They sit 4.8-5.9 pp below an oracle, so the bottleneck is whether a correct candidate exists.
• With Gemma, confidence argmax is worse than random

Evidence

• Calibration before/after (gpt-oss/math, 23,391 candidates)
  • ECE 0.278 to 0.008 with isotonic, Brier 0.302 to 0.208.
  • Temperature scaling (T=4.35) only reached ECE 0.142.
• Router comparison (9,653 blocks)
  • Expected success: argmax 0.497, random valid 0.490, confidence-weighted vote 0.501.
  • The oracle reaches 0.549.
• Cross-model 2x2 grid
  • Raw AUROC: gpt-oss math 0.721, bio 0.637, Gemma bio 0.537, math 0.440.
  • On Gemma math, argmax scores 0.362 versus 0.474 for random.
• Commitment shift and cost
  • Unconditional correctness shift is -1.7 pp on gpt-oss math and +0.9 to +12.2 pp in the other three cells.
  • Broadcast beats single_agent in pass rate (0.892 vs 0.788) but uses 12.8x tokens.

How to Apply

• Build a confidence reliability table from your logs
  • Today, collect agent responses with correctness labels and compute actual accuracy per confidence bin.
  • Also compute AUROC and mean confidence versus accuracy to check overconfidence.
• Calibrate probability-like scores with isotonic regression
  • If you threshold confidence for early stopping or human handoff, fit isotonic regression, e.g. with scikit-learn.
  • Split folds by question so no question appears in both fit and evaluation.
• A/B routing against random selection
  • On the same candidate blocks, compare confidence argmax with random valid selection.
  • If argmax is not better, turn off confidence routing.
• Log poll answers and final spoken answers separately
  • If your pipeline uses both structured and free-form outputs, extract both and record disagreement rate and correctness change.
  • If divergence hurts, experiment with locking the structured answer.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: auditing confidence scores)

I have LLM agent logs. Each row is (confidence 0-1, correct 0/1).

{{paste CSV data}}

Please compute:
1. Split confidence into 10 equal-width bins and give mean confidence and actual accuracy per bin
2. ECE and Brier score
3. AUROC (probability a correct answer gets higher confidence than a wrong one)
4. Gap between mean confidence and overall accuracy

Then write Python code that cross-fits isotonic regression with 2 folds split by question, and compares ECE and AUROC before and after calibration.

Next to try: ask for code comparing confidence argmax versus random selection on the same candidate blocks.

Terminology

Related Papers

• Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

  Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.

• Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents

  Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.

• Harness Engineering 101: The LLM is the brain, the harness is the body

  An agent is a loop managing a JSON array; 16 chapters build one from scratch

• Harness2 by Google Research – Recursive Agent Harnessing

  Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.

• Ousted Savannah police officer gave ICE agent husband access to Flock system

  A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired

• Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling

  Running Claude Code agents in tmux works well, but limits appear around four agents.