Cohen's Kappa is a statistical measure of agreement between exactly two raters labeling the same items into categories. It corrects for the agreement that would happen purely by chance, which raw percent agreement doesn't.

‍

Why it matters

If two annotators choose between two labels and guess at random, they'd agree about half the time when the labels are equally likely, and even more often when one label is far more common. A high raw agreement score can hide this, since guessing the common label produces high agreement without real skill. Kappa strips out that coincidental agreement, leaving a score that reflects genuine consistency between the two people.

‍

When kappa looks too low

Kappa can come out surprisingly low even when raw agreement is high, usually because one label is very rare. This is sometimes called the kappa paradox. It's why teams read kappa next to percent agreement instead of treating either number alone as the full story.

‍

What the score means

A score of 0 means agreement is no better than chance, and 1 means perfect agreement. Commonly cited rules of thumb treat values above roughly 0.6 as substantial and above 0.8 as almost perfect, though the right bar depends on the stakes, and these cut-offs are conventions, not laws. Because Cohen's version only handles two raters, projects with three or more move to Fleiss' Kappa or Krippendorff's Alpha.

‍