LLM Judge Bias
Systematic skew in judgments made by an LLM acting as an evaluator because of factors such as output order, length, phrasing, or persuasiveness that are weakly related to the evaluation criteria.
Example
When comparing two answers of similar substantive quality, the judge favors the answer shown first or the longer answer for reasons unrelated to the rubric.
Distinctions and cautions
LLM-as-a-Judge is an evaluation method; LLM Judge Bias is a source of distortion within such evaluation. Position and length are examples of factors to examine, not an exhaustive or universally fixed taxonomy of bias.
Evidence
View evidence →
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Title
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Author / organization
- Zheng et al.
- Year
- 2023
- Source type / publication status
- NeurIPS Datasets & Benchmarks; peer reviewed
- Supported concepts / claim context
- LLM-as-a-Judge; LLM Judge Bias; Human agreement; position, verbosity and self-enhancement biases.
- Scope and limitations
- MT-Bench, Chatbot Arena and GPT-4-era judges; early-generation judges; over-80% agreement is setting-specific.
- Access / version note
- Published proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Judging the Judges: A Systematic Study of Position Bias
- Title
- Judging the Judges: A Systematic Study of Position Bias
- Author / organization
- Shi et al.
- Year
- 2025
- Source type / publication status
- IJCNLP/AACL; peer reviewed
- Supported concepts / claim context
- LLM Judge Bias; Position bias varying by judge, task and answer-quality gap.
- Scope and limitations
- 15 judges, MTBench/DevBench, 22 tasks, about 40 generators and over 150k evaluations; position-bias subtype only.
- Access / version note
- Published proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Can You Trick the Grader? Adversarial Persuasion of LLM Judges
- Title
- Can You Trick the Grader? Adversarial Persuasion of LLM Judges
- Author / organization
- Hwang et al.
- Year
- 2025
- Source type / publication status
- Findings EMNLP; peer reviewed
- Supported concepts / claim context
- LLM Judge Bias; Grader Gaming boundary; Persuasive content inflating scores for incorrect solutions.
- Scope and limitations
- Six math benchmarks and seven persuasion techniques; adversarial rhetoric; not proof of spontaneous gaming.
- Access / version note
- Published Findings EMNLP.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Curse of Knowledge: … Biasing LLM Judges in Complex Evaluation
- Title
- Curse of Knowledge: … Biasing LLM Judges in Complex Evaluation
- Author / organization
- Li et al.
- Year
- 2025
- Source type / publication status
- Findings EMNLP; peer reviewed
- Supported concepts / claim context
- LLM Judge Bias; LLM-as-a-Judge; Auxiliary-information-induced bias and task complexity.
- Scope and limitations
- ComplexEval Bench with twelve basic and three advanced scenarios; benchmark-defined complex evaluation.
- Access / version note
- Published Findings EMNLP; abbreviated title as recorded in the source authority.
- Recorded official URL / DOI / identifier
- Official URL / recorded link