Preference ranking is the task of comparing two or more AI generated responses to the same prompt and ordering them from best to worst, based on which one a person or an AI evaluator actually prefers. Rather than asking someone to write a perfect response from scratch, it just asks them to judge which existing option is better.

‍

Why it matters

Judging which of two responses is better is usually much faster and more consistent than writing a full response yourself, since it's easier to recognize good quality than to produce it from a blank page. This is exactly why preference ranking became the core data collection method behind RLHF, it lets a team collect large volumes of useful feedback quickly, and that feedback is what trains a model to produce more of what people actually prefer.

‍

How the ranking task actually works

‍ Annotators are typically shown two or more responses to the identical prompt, without knowing which model or version produced which one, and asked to pick the better response or put them in order. Clear guidelines matter enormously here, since without them, different annotators can disagree about what better even means, one might prioritize thoroughness while another prioritizes brevity. Those guidelines usually spell out specific criteria to judge by, like helpfulness, accuracy, or tone, so rankings stay consistent across a large team of reviewers.

‍