Confident at the moment of action: belief miscalibration in LLM play under hidden information

Who Should Read

An agent developer who gates autonomous actions or human escalation on the model's own stated confidence score.

Core Mechanics

• High-confidence actions were almost always wrong
  • Captures at stated confidence ≥0.5 about the hidden royal piece (Regent) were correct in 1 of 62 cases across two batches.
  • What the model says it believes and what it does come apart at the moment of action.
• The error is concentrated in high-confidence moments
  • 99.3% of the Brier score (probability-error metric) gap came from high-confidence events, and 98.7% in the replication.
  • The model looks fine most of the time and fails exactly when it acts.
• Settings alone can move calibration a lot
  • For gemini-3-flash-preview, raising max_tokens from 4096 to 16384 lifted the high-confidence hit rate from 29.2% to 40.0%.
  • That is nearly as large as the 14.9-point gap between models 635.7 leaderboard points apart.
• The best on cost, speed and legality was worst on belief
  • With extended reasoning off, DeepSeek V4 Flash had 100% legal moves and was about 28x cheaper and 25x faster.
  • Its well-formed belief rate was only 3.8%.

Evidence

• High-confidence capture hit rate was 1/62
  • For S1 (gemini-3.1-flash-lite), captures at ≥0.5 confidence were 0/22 in the original batch and 1/40 in the replication.
  • Pooled this is 1.6%, with a 95% Wilson CI of [0.3%, 8.6%].
• Brier was 3.8–15.9x the uniform baseline
  • At capture time S1's Brier was 0.1445 versus 0.0203 for a matched uniform prior.
  • The 95% CI on the ratio was 3.8x–15.9x.
• Hit rates ordered across five seats
  • S1 was 1.6% (1/62), S7 (gpt-5-mini) 14.3% (2/14), S6-base 29.2% (7/24), S6B 40.0% (4/10), S5 50.0% (4/8).
  • Only the extremes (S1 vs S5) were statistically significant.
• 73.6% of captures hit a never-named square
  • Among 91 captures with two or more named candidates, the captured square was named in 24 (26.4%).
  • This was indistinguishable from a permutation null of 29.2% (p=0.218).

How to Apply

• Validate your confidence gate against real logs
  • Today, compute the actual accuracy of actions your agent took at confidence ≥0.5.
  • Start with tasks whose ground truth is confirmed later, and build a hit-rate table by confidence bin.
• Log belief and action as separate outputs
  • Ask for the model's probability estimate as its own field every turn, apart from the action.
  • Once the truth is known, compare the estimate to it to check calibration.
• Don't auto-execute on self-reported confidence alone
  • Add external verification before acting, such as rule checks, tests or tool lookups.
  • Redesign thresholds so higher confidence triggers more checking, not less.
• Re-measure calibration whenever settings change
  • Rerun the same calibration evaluation when you change max_tokens or reasoning mode.
  • Don't pick configurations on cost, speed and legality metrics alone.

Try It

# Paste into Claude/ChatGPT (simplified version of the paper's method: confidence calibration check)

Below is the log our agent produced right before executing actions.
{{log: each row has action, stated_confidence (0-1), and later-confirmed correct (true/false)}}

Please do the following.
1. Split stated_confidence into <0.5 / 0.5-0.8 / >0.8 and give a table of counts and actual hit rate per bin.
2. Compare each bin's Brier score to a baseline that always predicts the overall average accuracy.
3. If the hit rate in the high-confidence bin (≥0.5) is below baseline, suggest how to change our auto-execution threshold.

Next step: split the same log by model settings (temperature, max_tokens) and rerun.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.