A gold standard review is the process of checking a person's or a system's decisions against a gold standard dataset, comparing the answers directly to ones already known to be correct. Unlike comparing two reviewers to each other, this method has a definitive right answer to measure against.

‍

‍Why it matters

Comparing people to each other, like inter-annotator agreement or reviewer calibration, shows whether they're consistent, not whether the consistent answer is actually right. Two reviewers could confidently agree on the same wrong call, and agreement metrics alone would never catch it, since there's nothing in that comparison pointing to ground truth. A gold standard review closes that gap by checking against answers the team has already verified as correct.

‍

‍How it's usually run

‍This is usually done covertly, mixing gold standard items into someone's normal workload without telling them which ones are being checked, so the review measures their genuine, everyday judgment rather than their performance when they know they're being tested. A mismatch on a gold standard item counts differently than a disagreement with a peer, since it points to an actual error, not just a difference of opinion. The gold standard set itself typically gets rotated periodically, so people can't simply memorize the answers to items they recognize.