Annotation Guidelines That Hold Up: How to Write and Test Them

Andie Garcia
Andie Garcia
Marketing Manager
BUNCH Blog
>
Machine Learning
Last Update:
November 20, 2026
Annotation Guidelines: How to Write Them That Hold Up

A finished guideline can still produce inconsistent labels. Annotators may agree on easy items and split whenever the rules leave a hard case open to interpretation.

‍

Annotation guidelines define each label, when to apply it, and what to do when an item does not fit neatly. Test them in a pilot. For each recurring disagreement, identify the failed rule or example, revise it, and test again before scaling.

‍

At BUNCH, our clients typically provide a first draft of the labeling guidelines. We then maintain it day to day: we propose rules as new information and edge cases emerge, your policy owner approves them, and we train the whole team on the updated rules and shared examples. That keeps decisions consistent as the work evolves, and we use pilots and QA checks to verify whether the rules are working.

‍

What Annotation Guidelines Do

Two annotators read the same support ticket. One labels a sarcastic complaint negative; the other labels it positive. If the schema offers positive, negative, neutral, mixed, and unclear. The model then trains on conflicting labels.

‍

The fix is a decision test both annotators can use, not a request to be more careful.

‍

Here is how we would write that rule:

  • Name: Negative
  • Plain definition: The customer is expressing dissatisfaction
  • Inclusion criteria: Label Negative when the customer’s intended meaning is dissatisfaction, even if the words are positive
  • Exclusion criteria: Do not apply when the sarcasm is aimed at a third party rather than at the product or service
  • Edge case (ambiguous): If intent can’t be determined from the ticket alone, mark Unclear and escalate
  • Example: “Great, another outage.” → Negative, because the intended meaning is frustration

‍

ML leads own the schema, policy owners approve policy decisions, operations leads train reviewers, and annotators apply the rules. These labels become training data for supervised models.

A team lead explaining annotation guidelines to annotators

Could a new annotator label a hard case without asking you?

‍

The Four Parts Every Guideline Needs

For each label, specify:

  1. Name: A clear, unique label name that annotators can recognize at a glance. Avoid names that overlap with other labels or could be read in more than one way.
  2. Plain definition: What the label means, in simple words. If an annotator can’t explain the label in a sentence or two, the definition needs work.
  3. Inclusion criteria: What should be labeled. List the signals that qualify an item, and add a short example for each.
  4. Exclusion criteria: What should not be labeled. State what the label is not, especially for items that look similar to labels that do apply.

‍

Edge cases. Beyond the four parts, add a short section of tricky situations and how the guideline settles them. The infographic names two kinds:

  • Missing context: If key information is missing, use the available evidence and follow the default decision rule. Examples include a ticket with no earlier messages, a blurry image or clipped audio.
  • Ambiguous cases: If an item could fit multiple labels, apply the tie-breaker rule, such as choosing the most specific label. Add sarcasm, code-switching and reclaimed language here if your task includes them, and say when an item should be escalated instead of labeled.

‍

Make the rules specific to the task. For bounding boxes, say whether annotators must use the minimum enclosing rectangle or may leave padding, how to handle partially visible objects, and when an image is complete. For video, define how object identity carries across frames. For audio, cover overlapping speakers and clipped words.

‍

Four Parts of Annotation Guidelines

Measure Agreement and Correctness

Set an acceptance threshold before the pilot and save it with the guideline version. Report results by label and media type so a strong overall score cannot hide a weak category.

‍

MeasureWhat it showsBest use
Raw agreementShare of jointly labeled items with matching labelsFast pilot check
Cohen's kappaTwo-annotator agreement adjusted for chanceCategorical labels, including uneven class frequencies
Krippendorff's alphaAgreement with multiple annotators and incomplete assignmentsLarger panels
F1 against gold labelsBalance of precision and recall against an accepted referenceEvaluating machine-generated or assisted labels
Majority voteThe label selected by a specified share of annotatorsPanel decisions on subjective items

Raw agreement can look high when one label dominates. Kappa adjusts for chance agreement; alpha handles varying annotator assignments. Against gold labels, precision tracks false positives and recall tracks misses. F1 combines the two.

‍

For subjective categories, define the panel rule in advance. Schroeder, Roy and Kabbara (Findings of ACL 2025) labeled each item with five annotators and kept a label when at least three chose it so keep model suggestions out of the pilot. They also found that showing annotators LLM suggestions pulled their labels toward the model’s. A 3-of-5 decision has less consensus than a 5-of-5 decision; consider adjudicating close votes. See the distinction between matching labels and reliable decisions in our note on inter-annotator agreement.

‍

Write, Pilot, and Revise

Use the annotation agreement framework as a loop, not a one-time sign-off:

  1. Design: Define the schema and what the model must learn.
  2. Train: Give annotators the same rules and worked examples.
  3. Pilot: Have at least two annotators label a shared sample independently.
  4. Review: Compare results with the preset threshold, adjudicate disagreements, revise the guideline, and retest.

‍

Include hard cases in the pilot, but do not reuse training examples. Annotators should not discuss answers before labeling; otherwise, the score tests collaboration rather than the written rules. Track sample size, training, agreement, adjudications, and version changes.

‍

In production, we review each batch and feed recurring disagreements back into the guideline. The same cycle underpins the data annotation process we use with client teams.

‍

Write, Pilot and Revise Guidelines of Data Annotation

Diagnose Disagreement Before Changing the Rules

A low score points to a problem, not its cause. Check the error pattern before rewriting.

‍

PatternLikely causeNext step
One label has low agreementOverlapping category boundariesAdd inclusion, exclusion, and priority rules
Agreement falls across labelsTraining or task-design problemRetrain and test on a new sample
One annotator relabels the same item differentlyUnclear ruleClarify it and check intra-annotator agreement
One media type performs poorlyMissing media-specific ruleDefine the geometry or context rule and retest
Agreement is high but audits find errorsShared misunderstandingAdd expert-reviewed gold examples and rationale

A case from our own work. Our first version of “mixed” meant “both positive and negative.” Annotators began using it for anything unclear. We rewrote it to require one positive and one negative element, and added an Unclear label. That is the first row of the table above: the problem was an overlapping boundary, not careless annotators.

‍

Agreement is not correctness. Bassi et al. (Findings of ACL 2025) found that agreement alone did not show labels were correct, and recommended expert oversight and periodic quality audits. Use an error taxonomy, gold examples with rationales, and an escalation path for new cases. Log each change, its owner, the labels affected, and the result of the follow-up pilot.

‍

A split vote may reveal an unclear rule. A unanimous wrong answer may reveal a flawed schema.

‍

Decide Who Owns the Guidelines

If your labels depend on sensitive policy decisions or internal approvals, keep ownership of the rules in-house and use a partner for training and QA. Daily annotation management may suit a partner when volume grows and the team cannot consistently train reviewers, run pilots, and audit batches. Define who approves rules, who updates examples, and who retrains annotators after a change.

‍

Do not scale a weak guideline through any team. Fix recurring disagreements and verify the revised rules first.

‍

How long does a guideline take?

Until a representative pilot meets the threshold set beforehand.

‍

How often should it change?

When audits or batch reviews reveal a recurring gap, with every change versioned and tested.

‍

Can you reuse another team's guideline?

As a draft, then test it against your data and schema.

‍

Who approves a mid-project change?

The policy owner approves the rule; the team lead retrains annotators before they use it.

‍

At BUNCH, we build managed teams for data labeling, content moderation, and AI validation. If your team needs help training annotators and maintaining an edge-case log as volume grows, compare notes with us.

About the Author

Andie Garcia
Andie Garcia
Andie is a Marketing Manager at BUNCH, where she works closely with the trust & safety and operations teams to translate their frameworks for community, moderation, and support into practical guidance.

Stay in the Loop!

Subscribe to our newsletter and get the latest updates, exclusive content, and insights on Data Ops, Machine Learning, and emerging tech startups.

Related Content

Data Annotation Process: A Reliable Pipeline Guide

Learn how a reliable data annotation process works, from quality checks to scalable workflows. Build pipelines you can trust in 2026.

Video Annotation Services Explained and How to Choose

Learn what video annotation services do, from bounding boxes to tracking, plus workflow, QA and how to choose video annotation services for your model.

Audio Annotation Service How to Choose and Scale Quality

Learn what an audio annotation service does, how quality control works, and how to choose a vendor that can scale without sacrificing label quality.