OSIIX Library

AI Reliability Catalog

A reference guide to 48 terms concerning AI reliability, errors, evaluation, agent behavior and failure, and human or security-related controls. Each entry explains a term, illustrates its use, distinguishes it from nearby concepts, and identifies relevant evidence sources.

48 terms

About these groups. The six groups are a browsing aid. They are neither an industry-standard classification scheme nor an ontology defining formal hierarchical relationships between terms.

Reliability & Uncertainty

Reliability

The property of an AI system being able to perform its required functions consistently under the conditions, over the period, and for the uses for which it is intended.

Accuracy

The extent to which a prediction, calculation, estimate, answer, or similar output agrees with the correct answer, a benchmark value, or an accepted reference value.

Calibration

The extent to which a model's expressed confidence corresponds to the frequency with which its predictions or answers are actually correct.

Confidence

A probability, score, or natural-language statement through which a model expresses how likely it considers a prediction or answer to be correct.

Uncertainty Estimation

A family of methods for estimating or quantifying how uncertain a model's output is, using information such as probability distributions, differences across repeated outputs, semantic variation, or ensembles combining multiple outputs or methods.

Robustness

The property of maintaining required performance or behavior when inputs, operating conditions, data distributions, or prompt wording change in reasonable ways.

Response Consistency

The extent to which a system's responses or decisions agree across identical inputs, semantically equivalent inputs, repeated runs, or related turns in a conversation.

Nondeterminism

The property of outputs not being uniquely reproducible under the same logical input and nominal inference settings because of run-time or execution-environment conditions.

Stochasticity

The property of potentially producing multiple outputs under the same conditions because a model or system intentionally incorporates probabilistic choices, such as sampling.

Abstention

A decision or operational practice of withholding a prediction or answer—such as responding “I don't know” or declining to make a judgment—when uncertainty or decision risk is high.

Distribution Shift

A condition in which the distribution of data encountered in deployment or another evaluation differs from the distribution used for training or a reference evaluation.

Facts, Evidence & Retrieval

Hallucination

A condition in which model-generated content can appear plausible yet is incorrect, unsupported, or inconsistent when assessed against relevant evidence, supplied context, or external facts.

Factuality

The extent to which generated claims are supported as factual by external reality or reliable reference knowledge.

Faithfulness

The extent to which an output is consistent with, and legitimately supported by, the provided materials, context, or retrieved evidence.

Groundedness

The property that important claims in an output are supported by, and traceable to, specified evidence, source material, or context.

Grounding

A process or system-level mechanism that connects generation or answers to external evidence, retrieved information, or other materials so that claims can be based on that information.

Knowledge Conflict

A condition in which knowledge held internally by a model conflicts with context or external information supplied through a prompt or retrieval.

Retrieval Failure

A failure to obtain information needed for an answer or operation, or the retrieval of irrelevant or otherwise unsuitable information instead.

Citation / Evidence Attribution Quality

The quality of whether citations or evidence attached to a claim actually support that claim and correctly point to the relevant material.

Retrieval-Augmented Generation

A family of system designs and methods that retrieve information from external sources and provide it to a language model during generation of an answer or other text.

Verification

A process of examining a generated claim or answer through additional evidence, tests, comparisons, or another means of checking it.

Model Behavior & Failure

Prompt Sensitivity

The property of a model's output or task performance changing substantially in response to small surface-level differences—such as wording, punctuation, format, or order—between prompts with nearly identical meaning.

Long-context Utilization Failure

A failure to locate, select, or appropriately use required information in a long context even though that information fits within the model's intended context length.

Instruction-following Failure

A failure by a model to satisfy an explicitly stated task, style, format, situational requirement, or other constraint.

Sycophancy

A tendency or failure in which a model adjusts its answers or judgments to match a user's expressed belief, assertion, or desired conclusion rather than maintaining evidence-based, independent assessment.

Over-refusal

A failure in which a model unnecessarily refuses a harmless request that it could appropriately answer under the applicable policy and conditions of use.

Unsafe Compliance

A failure in which a model provides actionable assistance in response to a request that should be refused or safely constrained under the applicable policies or task design.

Reasoning Failure

A failure on a task requiring reasoning, in which a system fails at inference, decomposition, applying constraints, logic, arithmetic, synthesis, or related operations and reaches an incorrect conclusion, procedure, or decision.

Self-correction Failure

A failure arising when a model attempts or is asked to detect, reconsider, or correct an error in its own answer or reasoning but does not correct it, changes something that was already correct, or makes the result worse.

Evaluation & Benchmarks

LLM-as-a-Judge

A method of using an LLM as an evaluator to assess, compare, score, or rank outputs produced by models or systems.

LLM Judge Bias

Systematic skew in judgments made by an LLM acting as an evaluator because of factors such as output order, length, phrasing, or persuasiveness that are weakly related to the evaluation criteria.

Benchmark Contamination

An evaluation-validity problem in which benchmark or test material, or closely related information, enters the training or other pre-evaluation exposure of the system being tested.

Solution Contamination

An evaluation-validity problem in which a model or agent accesses an answer or answer-revealing information that would not be available under the intended realistic conditions, then uses it to complete the task during evaluation.

Grader Gaming

An evaluation problem in which a system produces outputs that obtain high scores by exploiting weaknesses in a grader's implementation or mismatches in proxy metrics, without substantively satisfying the intended task.

Agent Execution

AI Agent

An AI system that handles goal-relevant state and advances work through multiple interactions with an environment, tools, software, or other components, rather than relying solely on one model inference.

Planning Failure

A failure to appropriately construct, maintain, update, or revise the sequence, dependencies, or steps of actions intended to achieve a goal.

Tool-selection Failure

A failure to select an appropriate tool or action needed for a task, instead choosing something incorrect, unnecessary, or insufficient.

Tool-use Failure

A failure to appropriately invoke or use a selected tool, set arguments, follow an execution order, interpret results, or recover from errors.

Memory Failure

A failure by an agent or system to store, retrieve, update, maintain, invalidate, or use task-relevant information about earlier interactions or work state.

Agent Non-termination

A problem in which an agent or system does not reach an appropriate completion or stopping condition and continues unnecessary iterations or actions.

Error Propagation

A problem in which an error originating at one stage or component is carried into later reasoning, state, actions, tool operations, outputs, or evaluation.

Cascading Failure

A problem in which one failure triggers other failures across multiple stages, components, or failure types and spreads into broader downstream failure.

People, Control & Security

Goal Misalignment

A condition in which there is a meaningful mismatch between a goal specified by a person, user, or operator and the actions a system selects or the outcomes it produces.

Overreliance

A pattern in which a person accepts AI outputs or advice, or delegates judgment to them, beyond what is warranted by their accuracy, uncertainty, task conditions, and the need for checking.

Prompt Injection

A problem in which instructions embedded in untrusted input disrupt the instruction or control boundaries that a model or system is intended to follow.

Indirect Prompt Injection

Prompt injection delivered through a web page, email, file, search result, retrieved document, tool response, or similar material prepared by an attacker or untrusted third party, rather than being supplied by the user as the primary instruction.

Excessive Agency

A condition in which an AI system is granted or exercises capabilities, permissions, autonomous decision-making authority, or ability to affect external systems beyond what is necessary for a task and its risks.

Human Oversight

The roles and mechanisms through which people monitor, check, approve, return for revision, intervene in, stop, or override operations or decisions involving AI systems.