Solution Contamination
An evaluation-validity problem in which a model or agent accesses an answer or answer-revealing information that would not be available under the intended realistic conditions, then uses it to complete the task during evaluation.
Example
During a benchmark run, an agent retrieves an explanation published after the task was created and uses the published solution to complete the task.
Distinctions and cautions
Benchmark Contamination concerns exposure to benchmark material or similar information before evaluation, whereas Solution Contamination concerns inappropriate solution access during evaluation. It is also distinct from Grader Gaming, where a system exploits gaps in the scoring method. The distinction here is whether the system accessed answer information beyond the evaluation's intended conditions.
Evidence
View evidence →
Cheating On AI Agent Evaluations
- Title
- Cheating On AI Agent Evaluations
- Author / organization
- NIST CAISI
- Year
- 2025
- Source type / publication status
- Official technical publication; published
- Supported concepts / claim context
- Solution Contamination; Grader Gaming; explicit definitions; evaluation-log cases and mitigations.
- Scope and limitations
- Agentic coding/cyber evaluations; historical logs and reported percentages are lower bounds, not population rates.
- Access / version note
- Published official source.
- Recorded official URL / DOI / identifier
- Official URL / recorded link