Grader Gaming
An evaluation problem in which a system produces outputs that obtain high scores by exploiting weaknesses in a grader's implementation or mismatches in proxy metrics, without substantively satisfying the intended task.
Example
A grader checks only one portion of an output, and an answer passes by matching that portion even though it does not meet the actual requirements.
Distinctions and cautions
Grader Gaming does not presume human-like malicious intent. It differs from legitimate improvements against a scoring metric when those improvements continue to satisfy the intended task. It is also distinct from accessing unauthorized answer information (Solution Contamination) and bias in an LLM evaluator (LLM Judge Bias).
Terminology note
How Grader Gaming manifests depends on the design of the grading metrics and the evaluation environment.
Evidence
View evidence →
Cheating On AI Agent Evaluations
- Title
- Cheating On AI Agent Evaluations
- Author / organization
- NIST CAISI
- Year
- 2025
- Source type / publication status
- Official technical publication; published
- Supported concepts / claim context
- Solution Contamination; Grader Gaming; explicit definitions; evaluation-log cases and mitigations.
- Scope and limitations
- Agentic coding/cyber evaluations; historical logs and reported percentages are lower bounds, not population rates.
- Access / version note
- Published official source.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Can You Trick the Grader? Adversarial Persuasion of LLM Judges
- Title
- Can You Trick the Grader? Adversarial Persuasion of LLM Judges
- Author / organization
- Hwang et al.
- Year
- 2025
- Source type / publication status
- Findings EMNLP; peer reviewed
- Supported concepts / claim context
- LLM Judge Bias; Grader Gaming boundary; Persuasive content inflating scores for incorrect solutions.
- Scope and limitations
- Six math benchmarks and seven persuasion techniques; adversarial rhetoric; not proof of spontaneous gaming.
- Access / version note
- Published Findings EMNLP.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests
- Title
- Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests
- Author / organization
- Akter et al.
- Year
- 2026
- Source type / publication status
- Findings ACL; peer reviewed
- Supported concepts / claim context
- Grader Gaming; Evaluator/proxy gaming detection and mitigation.
- Scope and limitations
- Four tasks, two scales, two training methods, two judges and 1,200 human-annotated cases; equivalence between general proxy gaming and grader gaming is not universal.
- Access / version note
- Published Findings ACL.
- Recorded official URL / DOI / identifier
- Official URL / recorded link