Reviewer calibration is the ongoing state of multiple reviewers, labelers, moderators, or QA analysts applying the same standard consistently, the goal that a calibration session is specifically designed to produce and maintain over time.‍

‍

Why it matters

Even with clear written guidelines, two people can read the same instructions and land on different decisions, especially on edge cases the guidelines didn't specifically cover. Left unchecked, that gap widens as reviewers spend more time working independently without ever comparing notes, and it shows up as inconsistent outcomes for reasons that have nothing to do with the actual work being judged.

‍

‍How it's maintained

Calibration isn't a one-time achievement. Teams typically re-run calibration sessions on a set cadence, or whenever someone new joins, rather than assuming alignment holds indefinitely after one early session. Metrics like inter-annotator agreement, or an equivalent quality score, get tracked over time specifically to catch when calibration has quietly slipped.