Annotation accuracy measures how closely labeled or reviewed work matches the correct, true answer, usually expressed as a percentage. It's calculated by comparing a sample of completed work against a trusted reference, like a gold standard dataset, expert judgment, or a verified answer key.

‍

‍Why it matters

A model, a moderation decision, or a support outcome can only be as good as the judgment behind it, so if that judgment is wrong even a small percentage of the time, the error compounds downstream. Low accuracy is often invisible until it's already caused a problem, which makes it a far more expensive issue to catch late than to catch during the work itself. This is why accuracy gets tracked as an ongoing metric rather than a one-time check at the start.

‍

‍How it's measured

A sample of completed work gets pulled, often weighted toward higher-risk or more ambiguous categories, and compared against a trusted reference to calculate a match percentage. Accuracy gets tracked over time and by category, not just as a single overall number, so a slipping category doesn't hide inside a strong average. When accuracy on a category drops below an agreed threshold, the people responsible for it get retrained or the guideline for that category gets clarified.