LLM-as-a-Judge
A method of using an LLM as an evaluator to assess, compare, score, or rank outputs produced by models or systems.
Example
Two answers to the same question are given to a separate LLM along with evaluation criteria, and the LLM ranks which answer better meets those criteria.
Distinctions and cautions
LLM-as-a-Judge denotes an evaluation method or role, not an objective ground truth or human oversight. It is not synonymous with Verification generally, which may require external evidence or tests. The judging LLM itself may exhibit LLM Judge Bias.
Terminology note
The appropriateness and effectiveness of using an LLM as an evaluator depend on what is evaluated, the criteria, the comparison baseline, and the operating conditions, among other factors.
Evidence
View evidence →
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Title
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Author / organization
- Zheng et al.
- Year
- 2023
- Source type / publication status
- NeurIPS Datasets & Benchmarks; peer reviewed
- Supported concepts / claim context
- LLM-as-a-Judge; LLM Judge Bias; Human agreement; position, verbosity and self-enhancement biases.
- Scope and limitations
- MT-Bench, Chatbot Arena and GPT-4-era judges; early-generation judges; over-80% agreement is setting-specific.
- Access / version note
- Published proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Curse of Knowledge: … Biasing LLM Judges in Complex Evaluation
- Title
- Curse of Knowledge: … Biasing LLM Judges in Complex Evaluation
- Author / organization
- Li et al.
- Year
- 2025
- Source type / publication status
- Findings EMNLP; peer reviewed
- Supported concepts / claim context
- LLM Judge Bias; LLM-as-a-Judge; Auxiliary-information-induced bias and task complexity.
- Scope and limitations
- ComplexEval Bench with twelve basic and three advanced scenarios; benchmark-defined complex evaluation.
- Access / version note
- Published Findings EMNLP; abbreviated title as recorded in the source authority.
- Recorded official URL / DOI / identifier
- Official URL / recorded link