Inter-Annotator Agreement (IAA) measures how consistently independent annotators label the same data, using chance-corrected statistics such as Cohen’s kappa or Krippendorff’s alpha. High IAA evidences that labels reflect a shared, documented standard rather than individual opinion - which is why regulated AI programs treat IAA as a contractual QA deliverable, with thresholds above 0.90.
A label is a judgment. If two trained people given the same guidelines produce different labels, the dataset encodes noise - and a model trained on it inherits that noise invisibly. IAA quantifies the consistency of judgment, making data quality measurable before the model makes it expensive.
Cohen’s kappa for two annotators on categorical labels; Fleiss’ kappa for multiple annotators; Krippendorff’s alpha as the general workhorse (any number of annotators, missing data, multiple data types); pairwise F1 or IoU for structured outputs like spans, boxes and masks, where "agreement" needs geometric definition.
IAA becomes audit evidence when it is systematic: dual-annotate a defined sample per cohort, score agreement, adjudicate disagreements through a senior reviewer, log the guideline version, and report the trajectory. LabelFort contracts IAA above 0.90 - scored with Cohen’s kappa, with IoU/F1 for geometric tasks - and ships the per-cohort record in every Evidence Export, turning a statistic into a compliance artifact for EU AI Act Article 10 and internal model-risk review.
Low agreement is rarely lazy annotators. It is usually ambiguous ontology, underspecified guidelines, or genuinely hard edge cases that need adjudication rules. The fix is upstream: refine definitions, add decision examples, retrain - then re-measure. Silent low IAA is how "95% accurate" datasets fail in production.
Agreement between two annotators, corrected for the agreement expected by chance. κ = 1 is perfect; κ = 0 is no better than chance. Regulated annotation QA contracts above 0.90.
Read More
Reliability for any number of annotators, tolerant of missing data, across nominal, ordinal, interval and ratio labels - the general case where kappa handles exactly two.
Read More
A tamper-resistant record of every event in an AI system’s data and model lifecycle, kept automatically over the system’s lifetime.
Read More
Depends on the metric and task, but for regulated training data, chance-corrected agreement above 0.90 is the evidence-grade bar LabelFort contracts. Research corpora often accept 0.67–0.80; audited production data should not.
Enough per cohort to be statistically meaningful - commonly 5–10% of volume, more for small or high-risk cohorts. The sampling plan itself should be documented as part of QA evidence.
The Act does not name IAA specifically, but Article 10 requires documented, examined data governance and quality practices for annotation. Measured agreement is the accepted way to evidence label quality against that requirement.
A low score on one cohort inside an otherwise strong project is more useful information than a lower overall average would suggest; it points to exactly where the ontology or guideline is breaking down, rather than implying the whole dataset is uniformly weaker. The fix targets that cohort specifically, rather than a blanket retrain across everything.
IAA is scored per cohort, with audit trails for every action & evidence exports mapped to EU AI Act Articles 10 & 12. Experience this on your own data in an evidence-grade proof of concept.
Certifications & readiness
