94% accuracy in 9 days: how LabelFort labeled 12,000 news sentences to a six-type schema

The client builds Named Entity Recognition and information extraction models that read national news at scale and turn unstructured reporting into structured, queryable data. Their models depend on identifying - accurately - who is named, which organizations are involved, where events happen, and when.

LabelFort annotated and validated 72,000 named entity labels across 12,000 national news sentences within 9 days at 94% annotation accuracy, using an AI-assisted maker-checker workflow across 46 annotators.

The challenge

NER annotation for news text requires strict accuracy in both entity boundaries and entity type classification - even small inconsistencies reduce model performance. News sentences frequently contain multiple entities in a single sentence, long multi-word names, conflicts between GPE vs PLACE vs FAC categories, titles attached to person names, and overlapping entity contexts.

Maintaining consistency across 46 annotators and 72,000 entity label decisions required strict guideline adherence and checker validation on every label.

• Multiple entities per sentence with long multi-word organization and location names
• GPE vs PLACE vs FAC classification conflicts across news contexts
• Titles and honorifics attached to person names requiring exact-boundary rules
• Overlapping entity contexts requiring no-nesting enforcement
• Consistent labeling across 46 annotators within a 9-day delivery window

What we delivered

As per the annotation guidelines, annotators tagged only these six entity types:

• PERSON
• ORG
• GPE
• PLACE
• FAC
• DATE

All other entity types - money, time, events, products, percentages, and so on - were ignored.

Golden rules that kept 46 annotators consistent

• Exact text only - tag only the entity words; no titles, prepositions, or extra words.
• Smallest complete entity - tag the shortest phrase that fully identifies the entity.
• No nesting, no overlap - when one entity sits inside another, tag only the larger one.
• Multi-word entities stay together - multi-word names are never split.

GPE vs PLACE vs FAC decision rule:

1. Governed location → GPE
2. Natural location → PLACE
3. Man-made structure → FAC

Our approach

LabelFort ran the engagement as AI-assisted pre-annotation followed by maker tagging and checker QA on every entity.

• AI-assisted suggestions accelerated entity detection for high-frequency patterns.
• Makers tagged entities using strict boundary rules and guideline-based conflict resolution.
• Checkers verified boundary correctness, label selection, no overlap or nesting, and GPE vs PLACE vs FAC classification.
• Every entity passed checker review before delivery - not a sample.

Results

The engagement delivered a consistent, model-ready dataset on schedule - every figure measured on the platform and validated through maker-checker QA.

Two reporting notes. Label totals count entity label decisions - 12,000 sentences assessed against each of the six schema types - not counted entity spans. Accuracy is checker-phase agreement against the project guideline, not an inter-annotator agreement (kappa) score.

• 12,000 national news sentences annotated
• 72,000 entity label decisions validated across the six-type schema
• 94% annotation accuracy across the delivered labels
• 677 hours 44 minutes of total agent effort
• 00:03:23 average handle time per sentence · 00:00:34 per labeled field
• 46 annotators on a maker-checker AI workflow
• Six entity types applied consistently: PERSON, ORG, GPE, PLACE, FAC, DATE

Key insights

• Strict boundary rules significantly improve model learning and entity consistency.
• The largest confusion point in news NER is GPE vs PLACE vs FAC.
• Maker-checker validation reduces inter-annotator drift on long datasets.
• Multi-word entity consistency improves downstream information extraction accuracy.

Impact

• National news NER model training and evaluation
• Entity-based search and indexing
• Real-time news monitoring tools
• Knowledge graph and entity linking pipelines
• Automated extraction of organizations, people, locations, and dates from news content

Key takeaways

• Labeled and validated 72,000 entity label decisions across 12,000 sentences in 9 days at 94% annotation accuracy.
• Maker-checker QA put checker validation on every entity, holding 46 annotators to one standard.
• AI-assisted pre-labeling accelerated entity detection; golden rules resolved GPE vs PLACE vs FAC conflicts.
• Six-type schema with strict boundary rules produced a dataset ready for high-performance NER and information extraction models.