Confidence
A probability, score, or natural-language statement through which a model expresses how likely it considers a prediction or answer to be correct.
Aliases: Verbalized Confidence.
Example
An answer is accompanied by “90% confidence” or “I am quite confident.” The number or expression is an instance of confidence.
Distinctions and cautions
In LLMs, a probability-derived score and a self-reported statement of certainty are not necessarily the same thing. High confidence does not guarantee that an answer is correct.
Open questions and cautions
For models whose internal probabilities cannot be accessed, there is not yet a single agreed-upon external representation that is most appropriate to use as a measure of confidence.
Evidence
View evidence →
Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models
- Title
- Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models
- Author / organization
- Zhao et al.
- Year
- 2024
- Source type / publication status
- Findings ACL peer-reviewed research; published
- Supported concepts / claim context
- Calibration and Confidence; prompting effects.
- Scope and limitations
- Findings depend on the question-answering tasks and prompting methods examined.
- Access / version note
- Published ACL Findings.
- Recorded official URL / DOI / identifier
- DOI:
10.18653/v1/2024.findings-acl.515(DOI resolver)
MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs
- Title
- MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs
- Author / organization
- Liu et al.
- Year
- 2025
- Source type / publication status
- EMNLP peer-reviewed research; published
- Supported concepts / claim context
- Confidence, Calibration, verbal uncertainty.
- Scope and limitations
- Addresses natural-language expressions of uncertainty, distinct from token-level probabilities.
- Access / version note
- Published EMNLP.
- Recorded official URL / DOI / identifier
- DOI:
10.18653/v1/2025.emnlp-main.1505(DOI resolver)
Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations
- Title
- Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations
- Author / organization
- Ji et al.
- Year
- 2025
- Source type / publication status
- EMNLP peer-reviewed research; published
- Supported concepts / claim context
- Confidence; verbal versus semantic uncertainty; hallucination.
- Scope and limitations
- Mainly short-form answers. The generalizability of representation intervention is limited.
- Access / version note
- Published EMNLP.
- Recorded official URL / DOI / identifier
- DOI:
10.18653/v1/2025.emnlp-main.187(DOI resolver)
Bayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models
- Title
- Bayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models
- Author / organization
- Tonolini et al.
- Year
- 2024
- Source type / publication status
- Findings ACL peer-reviewed research; published
- Supported concepts / claim context
- Uncertainty Estimation; prompt uncertainty.
- Scope and limitations
- Mainly natural-language classification; uses a small labeled validation set.
- Access / version note
- Published ACL Findings.
- Recorded official URL / DOI / identifier
- DOI:
10.18653/v1/2024.findings-acl.728(DOI resolver)