Annotation workflows typically involve: (1) creating clear labeling guidelines (what counts as "positive sentiment"? what's the boundary of a "car" in a bounding box?), (2) training annotators on the guidelines, (3) annotating data with multiple annotators per example (for quality control), (4) measuring inter-annotator agreement (do annotators agree on labels?), and (5) resolving disagreements (through adjudication or majority vote). Low agreement often indicates ambiguous guidelines or genuinely ambiguous data.
For LLM alignment, annotation means comparing model responses: "Is response A or response B better for this prompt?" This preference annotation is particularly challenging because "better" is subjective, context-dependent, and culturally variable. Annotator demographics, expertise, and instructions all influence the resulting preference data, which in turn shapes model behavior. This is why alignment is often described as encoding the values of whoever writes the annotation guidelines.
Increasingly, AI models assist with annotation: pre-labeling data that humans then correct (faster than labeling from scratch), generating synthetic annotation data, or serving as additional annotators alongside humans. This creates an interesting feedback loop: AI helps label data that trains better AI. The risk is that AI-assisted annotation inherits the biases of the assisting model, so human oversight remains essential.