Cohen’s kappa (κ) is a statistic that measures agreement between two annotators while correcting for the agreement expected by chance. It is computed as κ = (pₒ − pₑ) / (1 − pₑ), where pₒ is observed agreement and pₑ is chance agreement. κ = 1 means perfect agreement; κ = 0 means agreement no better than chance. In regulated annotation QA, a threshold above 0.90 is the evidence-grade standard.
κ = (pₒ − pₑ) / (1 − pₑ). Here pₒ is the observed proportion of items on which two annotators agree, and pₑ is the proportion of agreement expected by chance, computed from each annotator’s label distribution. Correcting for chance is the point: two annotators randomly labeling a 90/10 class split will "agree" often - kappa removes that illusion.
Common reference bands (Landis & Koch): below 0.20 slight; 0.21–0.40 fair; 0.41–0.60 moderate; 0.61–0.80 substantial; 0.81–1.00 almost perfect. For training data that must survive audit, LabelFort contracts a threshold above 0.90, validated on your dataset during the PoC - because "substantial agreement" is not a phrase you want in front of a regulator.
Kappa turns annotation quality from an opinion into a number: score a sample of dual-annotated items per cohort, report κ alongside acceptance criteria, adjudicate disagreements, and retrain or refine guidelines when κ dips. That per-cohort record - with annotator identities and guideline versions - is precisely the QA evidence EU AI Act Article 10 asks you to document.
Kappa handles exactly two annotators and nominal categories, is sensitive to class prevalence (the "kappa paradox"), and says nothing about which annotator is right. For more than two annotators, ordinal labels, or missing data, use Krippendorff’s alpha; for ground-truth accuracy, use adjudicated gold sets.
Two annotators, 100 medical images, "lesion" or "no lesion" - traced end to end.
pₒ = 0.92
The annotators agree on 92 of 100 items.
pₑ = 0.51
(0.60 × 0.55) + (0.40 × 0.45).
κ ≈ 0.84
0.41 / 0.49 - "almost perfect".
Below 0.90
Refine guidelines, re-measure.
Cohort QA record - ophthalmology-batch-07
Two annotators, binary labels. Enter the confusion counts and get κ instantly. Runs entirely in your browser; no data leaves the page.
| n = - | A said “Yes” | A said “No” |
|---|---|---|
| B said “Yes” | ||
| B said “No” |
Cohen’s κ-
How consistently independent annotators label the same data. High IAA evidences that labels reflect a shared, documented standard rather than individual opinion.
Read More
Reliability for any number of annotators, tolerant of missing data, across nominal, ordinal, interval and ratio labels - the general case where kappa handles exactly two.
Read More
A tamper-resistant record of every event in an AI system’s data and model lifecycle, kept automatically over the system’s lifetime.
Read More
Above 0.80 is conventionally "almost perfect"; for regulated annotation QA, contract above 0.90. Below 0.60, treat guidelines or ontology as broken.
Kappa compares exactly two annotators on nominal labels. Krippendorff’s alpha generalises to any number of annotators, handles missing data, and supports nominal, ordinal, interval and ratio data.
Percent agreement ignores chance. On imbalanced classes, random labeling produces high raw agreement; kappa corrects for it, which is why auditors and reviewers expect chance-corrected statistics.
IAA scored per cohort, audit trails on every action, evidence exports mapped to EU AI Act Articles 10 & 12. See it on your own data in an evidence-grade PoC.
Certifications & readiness
