Reliability
The property of an AI system being able to perform its required functions consistently under the conditions, over the period, and for the uses for which it is intended.
Example
A system handles similar inquiries over time, but on one particular day incorrect answers suddenly become frequent or required operations fail repeatedly.
Distinctions and cautions
Reliability is not merely a high rate of correct answers or the production of the same answer every time. Accuracy, robustness, and consistency are among the relevant dimensions, but reliability is not synonymous with any one of them.
Evidence
View evidence →
Artificial Intelligence Risk Management Framework (AI RMF 1.0)
- Title
- Artificial Intelligence Risk Management Framework (AI RMF 1.0)
- Author / organization
- NIST
- Year
- 2023
- Source type / publication status
- Official technical publication; published
- Supported concepts / claim context
- Reliability, Accuracy, Robustness; trustworthiness characteristics.
- Scope and limitations
- An AI-wide risk framework, not an LLM-specific taxonomy. A revision was underway as of 2026.
- Access / version note
- Treat the published edition as the historical/technical baseline.
- Recorded official URL / DOI / identifier
- DOI:
10.6028/NIST.AI.100-1(DOI resolver)
The Language of Trustworthy AI: An In-Depth Glossary of Terms
- Title
- The Language of Trustworthy AI: An In-Depth Glossary of Terms
- Author / organization
- NIST
- Year
- 2023
- Source type / publication status
- Official technical publication; published
- Supported concepts / claim context
- Terminology harmonization; concepts near Reliability and Accuracy.
- Scope and limitations
- This glossary promotes a shared vocabulary; it is not the unique definition accepted by every research community.
- Access / version note
- Terminology-alignment baseline.
- Recorded official URL / DOI / identifier
- DOI:
10.6028/NIST.AI.100-3(DOI resolver)
WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Title
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Author / organization
- Koh et al.
- Year
- 2021
- Source type / publication status
- ICML peer-reviewed research; published
- Supported concepts / claim context
- Distribution Shift, Robustness.
- Scope and limitations
- A general machine-learning benchmark, not limited to LLMs.
- Access / version note
- Published conference proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Evaluating Model Robustness and Stability to Dataset Shift
- Title
- Evaluating Model Robustness and Stability to Dataset Shift
- Author / organization
- Subbaswamy et al.
- Year
- 2021
- Source type / publication status
- AISTATS peer-reviewed research; published
- Supported concepts / claim context
- Robustness, Distribution Shift.
- Scope and limitations
- Primarily predictive-model settings.
- Access / version note
- Published conference proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Evaluating large language models for accuracy incentivizes hallucinations
- Title
- Evaluating large language models for accuracy incentivizes hallucinations
- Author / organization
- Kalai et al.
- Year
- 2026
- Source type / publication status
- Nature peer-reviewed research; published
- Supported concepts / claim context
- Accuracy, Abstention, Hallucination; evaluation incentives.
- Scope and limitations
- Formal model and selected frontier-model case study. Utility depends on the use case.
- Access / version note
- Published Nature article.
- Recorded official URL / DOI / identifier
- DOI:
10.1038/s41586-026-10549-w(DOI resolver)