Inter-rater reliability, or IRR, describes how consistently different people rate or label the same material. The term comes from research and clinical fields such as psychology and medicine, and in data labeling it's used for the same basic question that inter-annotator agreement asks.

‍

Why it matters

IRR has been used for decades to check whether different clinicians or researchers rate the same patient or behavior the same way. IAA became the more common phrase once data labeling for machine learning grew into its own field. Knowing both names helps a team find relevant research and tools published under either one.

‍

Agreement versus reliability

In everyday use the two terms are often treated as the same. Measurement researchers draw a finer line. Agreement asks whether raters gave identical ratings. Reliability asks whether the ratings consistently tell items apart, even if raters don't land on the exact same value. The two can come apart. Two raters can rank items in the same order while one always scores higher, which is strong reliability but weak exact agreement. For numeric ratings, reliability is often measured with the intraclass correlation coefficient, while category labels usually use kappa or alpha.

‍

What it means for labeling teams

For category labels, the metrics are the ones used under the IAA name: percent agreement, Cohen's Kappa, Fleiss' Kappa, and Krippendorff's Alpha. The main practical difference is emphasis. Clinical IRR work leans on rater training and judgment, while machine learning teams tend to focus on how disagreement affects model quality.