Label drift is a change over time in the labels a model or labeling team works with. The term is used two ways. In machine learning monitoring, it means the mix of labels shifts, such as fraud going from a tiny share of transactions to a larger one. In annotation work, it also describes the meaning or correct use of a label slowly changing, so the same word ends up tagging different things than before.

‍

Why it matters

Both versions hurt for the same reason, the data a model learned from stops matching reality. A shifted label mix can make a model's decision thresholds wrong even though the relationship between inputs and labels hasn't changed. A shifting label meaning is quieter. A new annotator reads an edge case differently, or the content itself changes, like new slang in moderation, and older and newer labels quietly stop being consistent. Because both happen gradually, they often go unnoticed until a model trained on the newer data behaves inconsistently with the older data.

‍

Telling it apart from concept drift

Label drift is often confused with concept drift. In the label-mix sense, only the proportions of the labels change, while the relationship between inputs and labels stays the same. Concept drift means that relationship itself changes, so the same input should now get a different label. Which one a team is dealing with decides the fix, recalibrating thresholds for the first and relabeling or retraining for the second.

‍

How teams catch it

For a shifting label mix, teams monitor how often each label appears over time and compare recent periods against a historical baseline. For shifting label meaning, they compare inter-annotator agreement across time windows, this month against last month, and re-review older labeled batches against current guidelines. A sudden drop in agreement or a jump in one label's share is worth investigating before it reaches a model.