Instruction-following Failure
A failure by a model to satisfy an explicitly stated task, style, format, situational requirement, or other constraint.
Example
Asked to provide exactly three bullet points, the model instead produces one long paragraph without bullets.
Distinctions and cautions
Factual incorrectness and failure to follow instructions are different evaluation dimensions. An otherwise correct answer may still fail an explicit formatting requirement. Conversely, an intended safety refusal is not automatically an Instruction-following Failure.
Evidence
View evidence →
FollowBench
- Title
- FollowBench
- Author / organization
- Jiang et al.
- Year
- 2024
- Source type / publication status
- ACL; peer reviewed
- Supported concepts / claim context
- Instruction-following failure; five constraint categories and difficulty levels.
- Scope and limitations
- 13 open/closed LLMs; includes a benchmark-specific automatic judge component.
- Access / version note
- Published ACL.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
InFoBench
- Title
- InFoBench
- Author / organization
- Qin et al.
- Year
- 2024
- Source type / publication status
- Findings ACL; peer reviewed
- Supported concepts / claim context
- Instruction-following Failure; Following decomposed instruction requirements.
- Scope and limitations
- 500 instructions and 2,250 decomposed questions; findings depend on evaluator construction.
- Access / version note
- Published Findings ACL.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
LLM Self-Correction with DeCRIM
- Title
- LLM Self-Correction with DeCRIM
- Author / organization
- Ferraz et al.
- Year
- 2024
- Source type / publication status
- Findings EMNLP; peer reviewed
- Supported concepts / claim context
- Instruction-following Failure; Self-correction Failure; Multi-constraint failure; critic-assisted refinement; self-correction.
- Scope and limitations
- RealInstruct, IFEval, GPT-4/Mistral and related conditions; critic feedback means this is not pure intrinsic self-correction.
- Access / version note
- Published Findings EMNLP.
- Recorded official URL / DOI / identifier
- Official URL / recorded link