The Agreement Rate Illusion

High simple agreement rates can mask systematic disagreement when annotation categories are imbalanced.

In LLM annotation studies, models show 71–88% simple agreement with human annotators, but intercoder reliability (Krippendorff’s alpha) ranges only 0.12–0.41. The discrepancy: alpha corrects for chance agreement, which inflates simple agreement when one category dominates.

A model that always predicts the majority class achieves high agreement but provides no useful signal. This pattern appears repeatedly in real-world annotation tasks where categories are rarely balanced.

When evaluating annotator quality, simple agreement rates are insufficient. Chance-corrected metrics reveal what matters.

Related: [None yet]