Consensus review is a quality process where multiple annotators independently label the same item, and their answers are compared to determine a final label based on where the majority or all of them agree. When reviewers disagree, the conflict typically gets resolved through discussion or escalation rather than just picking one answer at random.

‍

‍Why it matters

A single annotator's judgment, even from someone well trained, can be wrong or influenced by a bad day, unclear guidelines, or a genuinely ambiguous case. Consensus review reduces that risk by requiring agreement from more than one person before a label is treated as final, which matters most for data that will become part of a gold standard or training set, where a single mistake can ripple outward into everything measured against it later.

‍

‍How it typically works

A common setup has two or three annotators label the same item independently without seeing each other's answers, and if they agree, that label gets finalized immediately. When they disagree, the item gets escalated to a senior reviewer or a group discussion, and the resolution often gets documented and folded back into the guidelines so the same disagreement doesn't keep recurring across the wider dataset.