Benchmark Contamination
An evaluation-validity problem in which benchmark or test material, or closely related information, enters the training or other pre-evaluation exposure of the system being tested.
Example
A model scores highly on questions nearly identical to a previously released test set but performs substantially worse on comparable questions it has not seen before.
Distinctions and cautions
Benchmark Contamination concerns exposure to test material or close information before the evaluation. It differs from Solution Contamination, where an evaluated model or agent accesses answers it should not have during the evaluation itself. Training on material from the same subject area does not automatically establish benchmark contamination.
Evidence
View evidence →
Artificial Intelligence Technology Evaluation (AITE)
- Title
- Artificial Intelligence Technology Evaluation (AITE)
- Author / organization
- NIST
- Year
- 2026
- Source type / publication status
- Official evaluation program; active
- Supported concepts / claim context
- Benchmark Contamination; Sequestered evaluation data to mitigate train/test contamination.
- Scope and limitations
- VLM tasks in a NIST testbed; initial tasks are not LLM text benchmarks.
- Access / version note
- Current program notice in supplied input.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
NIST AI 800-3: Expanding the AI Evaluation Toolbox with Statistical Models
- Title
- NIST AI 800-3: Expanding the AI Evaluation Toolbox with Statistical Models
- Author / organization
- Keller et al., NIST
- Year
- 2026
- Source type / publication status
- Official technical publication; published
- Supported concepts / claim context
- cross-cutting; Measurement validity; fixed-benchmark versus generalized accuracy.
- Scope and limitations
- 22 frontier API LLMs, three benchmarks plus simulation; not specifically a contamination or judge-bias study.
- Access / version note
- Published official source.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
ConStat: Performance-Based Contamination Detection in Large Language Models
- Title
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Author / organization
- Dekoninck et al.
- Year
- 2024
- Source type / publication status
- NeurIPS; peer reviewed
- Supported concepts / claim context
- Benchmark Contamination; Benchmark-specific non-generalizing performance uplift; contamination detection.
- Scope and limitations
- Diverse benchmarks/architectures and over 40 models; operational definition differs from pure training-data overlap.
- Access / version note
- Published proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- Title
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- Author / organization
- Zhang et al.
- Year
- 2024
- Source type / publication status
- NeurIPS Datasets & Benchmarks; peer reviewed
- Supported concepts / claim context
- Benchmark Contamination; GSM8k/GSM1k performance and evidence of overfitting in some model families.
- Scope and limitations
- Leading open/closed LLMs; many frontier models showed minimal signs, so this is not universal.
- Access / version note
- Published proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
MMLU-CF
- Title
- MMLU-CF
- Author / organization
- Zhao et al.
- Year
- 2025
- Source type / publication status
- ACL; peer reviewed
- Supported concepts / claim context
- Benchmark Contamination; Closed test sets and decontamination; score/ranking shifts.
- Scope and limitations
- Over 40 mainstream LLMs; multiple-choice and world-knowledge emphasis.
- Access / version note
- Published ACL.
- Recorded official URL / DOI / identifier
- Official URL / recorded link