Inter-annotator agreement, or IAA, is a general term for how much multiple people labeling the same data agree with each other, measured with a statistic chosen to fit the situation. It's the umbrella behind more specific measures like percent agreement, Cohen's Kappa, Fleiss' Kappa, and Krippendorff's Alpha, not a single fixed calculation.

‍

Why it matters

Agreement matters because when trained annotators can't agree on how to label the same data, the task itself is usually poorly defined, not the people. Low agreement is one of the clearest early warning signs that guidelines need clarifying before a project scales, since whatever causes disagreement between two annotators now will cause the same problem across a larger team later.

‍

Where it gets scrutinized hardest

Projects where a wrong label has serious consequences, such as medical or legal labeling, usually set the highest agreement bar before data is trusted, since a disagreement there can mean a misdiagnosed condition or a misclassified document, not just a mislabeled photo. Content moderation sits in the middle, where disagreement on obvious violations is rare but climbs fast on culturally or contextually ambiguous content.

‍

Which metric to use

Percent agreement is simple but ignores chance. Cohen's Kappa corrects for chance but handles only two raters. Fleiss' Kappa handles more than two raters with categorical labels, and Krippendorff's Alpha adds other data types and missing ratings. Teams usually check raw agreement first to spot obvious problems, then move to a chance-corrected metric for anything that has to stand up to scrutiny. Strictly speaking, agreement is about identical ratings and reliability is about consistent differentiation, a distinction covered under inter-rater reliability.