Fleiss' Kappa extends the idea behind Cohen's Kappa to three or more raters labeling items into categories, correcting for chance agreement across the whole group instead of a pair.

‍

Why it matters

Cohen's Kappa works cleanly only with exactly two raters, which is a real limit once a project has a larger review team. Fleiss' Kappa closes that gap, and it doesn't require the same people to rate every item, which helps when reviewer assignments shift day to day. It does expect every item to receive the same number of ratings.

‍

How it compares to other multi-rater options

It's limited to categorical labels, meaning distinct categories rather than rankings or scores, which is where it differs from Krippendorff's Alpha. Alpha handles other data types, missing ratings, and a varying number of raters. Teams usually pick Fleiss' Kappa when labels are simple categories, every item gets the same number of ratings, and there are more than two raters, and move to Krippendorff's Alpha when the data is messier.