Krippendorff’s alpha (α) is a reliability coefficient that measures agreement among any number of annotators, tolerates missing data, and works across nominal, ordinal, interval and ratio data. It is computed as α = 1 − (observed disagreement / expected disagreement). Use α when you have more than two annotators, incomplete overlap, or non-categorical labels - the situations where Cohen’s kappa does not apply.
Real annotation programs rarely look like the textbook two-annotators-label-everything setup. Cohorts rotate, items get different numbers of judgments, labels can be ordinal (severity 1–5) or continuous. Alpha handles all of it in one framework, which makes it the natural companion to kappa in a rigorous QA methodology. LabelFort’s contracted metric is Cohen’s kappa, with IoU/F1 for geometric tasks; alpha is applied where task structure requires it.
Alpha compares observed disagreement among all annotator pairs on all items against the disagreement expected if labels were assigned at random, using a difference function appropriate to the data type (nominal, ordinal, interval, ratio). α = 1 means perfect reliability; α = 0 means chance-level; negative values mean systematic disagreement.
Krippendorff’s own guidance: rely on data with α ≥ 0.80; draw only tentative conclusions from 0.667 ≤ α < 0.80. For regulated training data, evidence-grade annotation programs hold cohorts to a higher bar - above 0.90 wherever task structure permits.
Agreement between two annotators, corrected for the agreement expected by chance. κ = 1 is perfect; κ = 0 is no better than chance. Regulated annotation QA contracts above 0.90.
Read More
How consistently independent annotators label the same data. High IAA evidences that labels reflect a shared, documented standard rather than individual opinion.
Read More
A tamper-resistant record of every event in an AI system’s data and model lifecycle, kept automatically over the system’s lifetime.
Read More
Use alpha with more than two annotators, missing or unbalanced judgments, or ordinal/interval/ratio labels. Use kappa for the simple two-annotator nominal case - it is more widely recognized there.
Alpha at or above 0.80 for reliable conclusions per Krippendorff. Evidence-grade annotation programs target higher, above 0.90, where the task permits.
Not directly, since they're computed differently and don't convert on a fixed formula. In practice, a dataset with a strong kappa score under the two-annotator nominal setup will usually also show a strong alpha score if recomputed under alpha's framework, but the two aren't interchangeable numbers; they're answering the same underlying question of reliability with different tolerances for the shape of the data.
Missing judgments don't need to be imputed or treated as disagreements; alpha's expected disagreement calculation accounts for how much overlap actually exists between annotators, rather than assuming every item received a judgment from every annotator. This is the specific property that makes alpha usable in real annotation workflows, where perfect overlap across a rotating team is rarely practical.
IAA is scored per cohort, with audit trails for every action & evidence exports mapped to EU AI Act Articles 10 & 12. Experience this on your own data in an evidence-grade proof of concept.
Certifications & readiness
