Distribution Shift
A condition in which the distribution of data encountered in deployment or another evaluation differs from the distribution used for training or a reference evaluation.
Example
A model evaluated mainly on technical documents is deployed in an environment with very different vocabulary and question patterns, and its answer accuracy declines.
Distinctions and cautions
Distribution Shift refers to the changed data or environmental conditions themselves. Whether the system maintains required performance under those changes is a question of Robustness. How far performance generalizes outside the previous distribution is a separate evaluation question.
Open questions and cautions
In operational LLM use, the extent to which changes in users, topics, time, prompts, or connected tools should count as changes in the “distribution” depends on the intended use.
Evidence
View evidence →
WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Title
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Author / organization
- Koh et al.
- Year
- 2021
- Source type / publication status
- ICML peer-reviewed research; published
- Supported concepts / claim context
- Distribution Shift, Robustness.
- Scope and limitations
- A general machine-learning benchmark, not limited to LLMs.
- Access / version note
- Published conference proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link
Evaluating Model Robustness and Stability to Dataset Shift
- Title
- Evaluating Model Robustness and Stability to Dataset Shift
- Author / organization
- Subbaswamy et al.
- Year
- 2021
- Source type / publication status
- AISTATS peer-reviewed research; published
- Supported concepts / claim context
- Robustness, Distribution Shift.
- Scope and limitations
- Primarily predictive-model settings.
- Access / version note
- Published conference proceedings.
- Recorded official URL / DOI / identifier
- Official URL / recorded link