Using LLM for confidence scoring is a bad idea

Oct 21, 2026•jimauthors•View Original

TL;DR Highlight

An LLM's self-reported confidence is plausible text, not a measurement; build scores from outside rules.

Who Should Read

A backend developer who routes agent replies to humans based on confidence scores the LLM wrote itself

Core Mechanics

A 0.85 from an LLM is not a measurement
- An LLM writes tokens one at a time, and the number is written like any other text.
- It is not read off an internal gauge but is a plausible continuation.
Trustworthy confidence needs two properties
- It must reflect how much information the prediction is based on.
- It must be checked against real outcomes on the same kind of question.
LLM scores meet neither property
- Pretraining scores next-token prediction and never checks whether written confidence matches accuracy.
- The model learned what confidence scores look like in text.
You can see it yourself
- Asking again can turn 0.85 into 0.7.
- Rewording to "Are you sure?" moves the score, and a prior "clearly the right answer" pulls it high.
Build confidence with a rule outside the LLM
- The LLM writes the answer and the score comes from observable information.
- Examples are whether search found articles, how many, whether the question type was seen before, and whether two replies agree.
No record, no score
- Save the information, score, and correctness for every reply, then update the rule from actual rates per group.
- Make a weekly table of actual correctness per confidence level to catch overconfidence.

Community Reactions

A view that LLMs can express confidence, but the asking was the problem
- Instead of a 0-100 scale, split into several axes with small enums or binary choices and weight them, the comment argues.
A rebuttal that training can calibrate it
- RLCR adds a Brier penalty to the reward so confidence is scored, not just correctness.
- A worked example shows a model that tells easy from hard questions gets a lower penalty (0.15 vs 0.24) than one always saying 0.6.
Practitioner experience with logprobs and classic ML
- Commenters suggested fixing the output classes and reading logprobs, or sampling the same prompt many times to measure choice distribution.
- One reported good results with BERT, pooling and a probability-calibrated SVM, noting it needs training data.
Experience that it was unreliable in real products
- One commenter joined a company where an LLM grades text with a confidence interval and found the outputs unreliable.
- Others reported fine-tuning confidence on their domain, or using grading scripts and A-to-F rubrics with explicit criteria.

How to Apply

Test your LLM score's consistency today
- Run your real confidence prompt on 10 cases, 5 times each, and check the score spread.
- If scores wobble on identical input, stop routing on them.
Log three observable signals
- Store retrieved document count, whether two answers to the same question agree, and the question type for each reply.
- Use these instead of the LLM's self-rating in the score slot.
Collect review outcomes as correctness labels
- Record in a correct field whether a human approved or the customer asked again.
- Build a per-signal-combination accuracy table as your first rule.
Make a weekly calibration table
- Compute actual correctness per confidence bucket.
- If the 90% bucket is right only 70% of the time, lower the rule and raise the human-review threshold.

Try It

copy & run

# Paste into Python: a simple calibration table (standard library only)

from collections import defaultdict

# Reply log: (article found, two answers agree, was correct)
records = [
    (True, True, True), (True, True, True), (True, False, False),
    (False, False, False), (False, True, False), (True, True, True),
]

groups = defaultdict(list)
for has_doc, agree, correct in records:
    groups[(has_doc, agree)].append(correct)

base = sum(c for *_, c in records) / len(records)
print(f'기본 정답률 {base:.0%}')
for key, vals in groups.items():
    rate = sum(vals) / len(vals)
    print(key, f'n={len(vals)}', f'confidence={rate:.0%}')

# Next: swap records for your real review log and compare weekly against per-bucket accuracy

Terminology

CalibrationWhen predictions of 70% come true about 70% of the time. A forecaster who says 70% on days when it rains 70% of the time is calibrated.

AutoregressiveGenerating text by picking each next token from the tokens before it. The confidence number is just more written text.

Brier penaltyThe squared gap between predicted probability and the outcome (1 if right, 0 if wrong). A comment cites it as what RLCR adds to the reward.

RLCRA training method that adds a Brier penalty to the reward so both correctness and confidence accuracy are learned. Introduced in an article a commenter linked.

logprobsThe log probabilities a model assigns to tokens. Commenters mentioned them as a relative confidence measure over fixed output classes.

Related Resources

Using LLM for confidence scoring is a bad idea

Oct 21, 2026•jimauthors•View Original

TL;DR Highlight

An LLM's self-reported confidence is plausible text, not a measurement; build scores from outside rules.

Who Should Read

A backend developer who routes agent replies to humans based on confidence scores the LLM wrote itself

Core Mechanics

A 0.85 from an LLM is not a measurement
- An LLM writes tokens one at a time, and the number is written like any other text.
- It is not read off an internal gauge but is a plausible continuation.
Trustworthy confidence needs two properties
- It must reflect how much information the prediction is based on.
- It must be checked against real outcomes on the same kind of question.
LLM scores meet neither property
- Pretraining scores next-token prediction and never checks whether written confidence matches accuracy.
- The model learned what confidence scores look like in text.
You can see it yourself
- Asking again can turn 0.85 into 0.7.
- Rewording to "Are you sure?" moves the score, and a prior "clearly the right answer" pulls it high.
Build confidence with a rule outside the LLM
- The LLM writes the answer and the score comes from observable information.
- Examples are whether search found articles, how many, whether the question type was seen before, and whether two replies agree.
No record, no score
- Save the information, score, and correctness for every reply, then update the rule from actual rates per group.
- Make a weekly table of actual correctness per confidence level to catch overconfidence.

Community Reactions

A view that LLMs can express confidence, but the asking was the problem
- Instead of a 0-100 scale, split into several axes with small enums or binary choices and weight them, the comment argues.
A rebuttal that training can calibrate it
- RLCR adds a Brier penalty to the reward so confidence is scored, not just correctness.
- A worked example shows a model that tells easy from hard questions gets a lower penalty (0.15 vs 0.24) than one always saying 0.6.
Practitioner experience with logprobs and classic ML
- Commenters suggested fixing the output classes and reading logprobs, or sampling the same prompt many times to measure choice distribution.
- One reported good results with BERT, pooling and a probability-calibrated SVM, noting it needs training data.
Experience that it was unreliable in real products
- One commenter joined a company where an LLM grades text with a confidence interval and found the outputs unreliable.
- Others reported fine-tuning confidence on their domain, or using grading scripts and A-to-F rubrics with explicit criteria.

How to Apply

Test your LLM score's consistency today
- Run your real confidence prompt on 10 cases, 5 times each, and check the score spread.
- If scores wobble on identical input, stop routing on them.
Log three observable signals
- Store retrieved document count, whether two answers to the same question agree, and the question type for each reply.
- Use these instead of the LLM's self-rating in the score slot.
Collect review outcomes as correctness labels
- Record in a correct field whether a human approved or the customer asked again.
- Build a per-signal-combination accuracy table as your first rule.
Make a weekly calibration table
- Compute actual correctness per confidence bucket.
- If the 90% bucket is right only 70% of the time, lower the rule and raise the human-review threshold.

Try It

copy & run

# Paste into Python: a simple calibration table (standard library only)

from collections import defaultdict

# Reply log: (article found, two answers agree, was correct)
records = [
    (True, True, True), (True, True, True), (True, False, False),
    (False, False, False), (False, True, False), (True, True, True),
]

groups = defaultdict(list)
for has_doc, agree, correct in records:
    groups[(has_doc, agree)].append(correct)

base = sum(c for *_, c in records) / len(records)
print(f'기본 정답률 {base:.0%}')
for key, vals in groups.items():
    rate = sum(vals) / len(vals)
    print(key, f'n={len(vals)}', f'confidence={rate:.0%}')

# Next: swap records for your real review log and compare weekly against per-bucket accuracy

Terminology

CalibrationWhen predictions of 70% come true about 70% of the time. A forecaster who says 70% on days when it rains 70% of the time is calibrated.

AutoregressiveGenerating text by picking each next token from the tokens before it. The confidence number is just more written text.

Brier penaltyThe squared gap between predicted probability and the outcome (1 if right, 0 if wrong). A comment cites it as what RLCR adds to the reward.

RLCRA training method that adds a Brier penalty to the reward so both correctness and confidence accuracy are learned. Introduced in an article a commenter linked.

logprobsThe log probabilities a model assigns to tokens. Commenters mentioned them as a relative confidence measure over fixed output classes.

Using LLM for confidence scoring is a bad idea

TL;DR Highlight

Who Should Read

Core Mechanics

Community Reactions

How to Apply

Try It

Terminology

Related Papers

Related Resources

Using LLM for confidence scoring is a bad idea

TL;DR Highlight

Who Should Read

Core Mechanics

Community Reactions

How to Apply

Try It

Terminology

Related Papers

Related Resources