Calibration
The extent to which a model's expressed confidence corresponds to the frequency with which its predictions or answers are actually correct.
Aliases: Confidence Calibration.
Example
Among many answers assigned 80% confidence, approximately 80% turn out to be correct. Such results may indicate good calibration.
Distinctions and cautions
Calibration concerns the relationship between confidence and actual correctness, not confidence alone. For an LLM, an evaluation can differ depending on whether confidence means token probabilities, an answer-level score, or a verbal expression of certainty.
Open questions and cautions
There is not yet a single settled answer to which unit of confidence should be matched against actual correctness in long, free-form responses.
Evidence
View evidence →
On Calibration of Modern Neural Networks
- Title
- On Calibration of Modern Neural Networks
- Author / organization
- Guo, Pleiss, Sun & Weinberger
- Year
- 2017
- Source type / publication status
- ICML peer-reviewed research; published
- Supported concepts / claim context
- Calibration; confidence versus correctness; temperature scaling.
- Scope and limitations
- Primarily studies neural networks for classification. Results cannot be directly generalized to verbal confidence in LLMs.
- Access / version note
- Published conference proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs
- Title
- MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs
- Author / organization
- Liu et al.
- Year
- 2025
- Source type / publication status
- EMNLP peer-reviewed research; published
- Supported concepts / claim context
- Confidence, Calibration, verbal uncertainty.
- Scope and limitations
- Addresses natural-language expressions of uncertainty, distinct from token-level probabilities.
- Access / version note
- Published EMNLP.
- Recorded official URL / DOI / identifier
- DOI:
10.18653/v1/2025.emnlp-main.1505(DOI resolver)
Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations
- Title
- Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations
- Author / organization
- Ji et al.
- Year
- 2025
- Source type / publication status
- EMNLP peer-reviewed research; published
- Supported concepts / claim context
- Confidence; verbal versus semantic uncertainty; hallucination.
- Scope and limitations
- Mainly short-form answers. The generalizability of representation intervention is limited.
- Access / version note
- Published EMNLP.
- Recorded official URL / DOI / identifier
- DOI:
10.18653/v1/2025.emnlp-main.187(DOI resolver)
Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models
- Title
- Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models
- Author / organization
- Zhao et al.
- Year
- 2024
- Source type / publication status
- Findings ACL peer-reviewed research; published
- Supported concepts / claim context
- Calibration and Confidence; prompting effects.
- Scope and limitations
- Findings depend on the question-answering tasks and prompting methods examined.
- Access / version note
- Published ACL Findings.
- Recorded official URL / DOI / identifier
- DOI:
10.18653/v1/2024.findings-acl.515(DOI resolver)