Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient and Reliable Estimation of Named Entity Linking Quality: A Case Study on GutBrainIE

Marco Martinelli, Stefano Marchesin, Gianmaria Silvello

TL;DR

The paper addresses the challenge of reliably estimating Named Entity Linking quality at scale when expert annotations are costly. It proposes a constrained optimization approach using Stratified Two-Way Cluster Sampling (STWCS) to efficiently estimate corpus-level NEL accuracy with formal statistical guarantees, demonstrated on the GutBrainIE corpus. In a case study, the framework achieved an overall accuracy of $0.915 \pm 0.0473$ while annotating only $24.6\%$ of triples and reduced annotation time by roughly $29\%$ compared to a simple random sampling baseline. The methodology is generic and applicable to other NEL benchmarks and IE pipelines, offering both corpus-level and strata-level insights and enabling targeted curation with reduced expert effort.

Abstract

Named Entity Linking (NEL) is a core component of biomedical Information Extraction (IE) pipelines, yet assessing its quality at scale is challenging due to the high cost of expert annotations and the large size of corpora. In this paper, we present a sampling-based framework to estimate the NEL accuracy of large-scale IE corpora under statistical guarantees and constrained annotation budgets. We frame NEL accuracy estimation as a constrained optimization problem, where the objective is to minimize expected annotation cost subject to a target Margin of Error (MoE) for the corpus-level accuracy estimate. Building on recent works on knowledge graph accuracy estimation, we adapt Stratified Two-Stage Cluster Sampling (STWCS) to the NEL setting, defining label-based strata and global surface-form clusters in a way that is independent of NEL annotations. Applied to 11,184 NEL annotations in GutBrainIE -- a new biomedical corpus openly released in fall 2025 -- our framework reaches a MoE $\leq 0.05$ by manually annotating only 2,749 triples (24.6%), leading to an overall accuracy estimate of $0.915 \pm 0.0473$. A time-based cost model and simulations against a Simple Random Sampling (SRS) baseline show that our design reduces expert annotation time by about 29% at fixed sample size. The framework is generic and can be applied to other NEL benchmarks and IE pipelines that require scalable and statistically robust accuracy assessment.

Efficient and Reliable Estimation of Named Entity Linking Quality: A Case Study on GutBrainIE

TL;DR

The paper addresses the challenge of reliably estimating Named Entity Linking quality at scale when expert annotations are costly. It proposes a constrained optimization approach using Stratified Two-Way Cluster Sampling (STWCS) to efficiently estimate corpus-level NEL accuracy with formal statistical guarantees, demonstrated on the GutBrainIE corpus. In a case study, the framework achieved an overall accuracy of while annotating only of triples and reduced annotation time by roughly compared to a simple random sampling baseline. The methodology is generic and applicable to other NEL benchmarks and IE pipelines, offering both corpus-level and strata-level insights and enabling targeted curation with reduced expert effort.

Abstract

Named Entity Linking (NEL) is a core component of biomedical Information Extraction (IE) pipelines, yet assessing its quality at scale is challenging due to the high cost of expert annotations and the large size of corpora. In this paper, we present a sampling-based framework to estimate the NEL accuracy of large-scale IE corpora under statistical guarantees and constrained annotation budgets. We frame NEL accuracy estimation as a constrained optimization problem, where the objective is to minimize expected annotation cost subject to a target Margin of Error (MoE) for the corpus-level accuracy estimate. Building on recent works on knowledge graph accuracy estimation, we adapt Stratified Two-Stage Cluster Sampling (STWCS) to the NEL setting, defining label-based strata and global surface-form clusters in a way that is independent of NEL annotations. Applied to 11,184 NEL annotations in GutBrainIE -- a new biomedical corpus openly released in fall 2025 -- our framework reaches a MoE by manually annotating only 2,749 triples (24.6%), leading to an overall accuracy estimate of . A time-based cost model and simulations against a Simple Random Sampling (SRS) baseline show that our design reduces expert annotation time by about 29% at fixed sample size. The framework is generic and can be applied to other NEL benchmarks and IE pipelines that require scalable and statistically robust accuracy assessment.
Paper Structure (20 sections, 11 equations, 1 figure, 3 tables)

This paper contains 20 sections, 11 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Task Setting
  3. Sampling and Accuracy Estimation in KG
  4. Named Entity Linking for GutBrainIE
  5. Accuracy Estimation for NEL Annotations
  6. Sampling Design
  7. Strata.
  8. Clusters.
  9. STWCS procedure.
  10. Estimator and Confidence Interval
  11. Cost Model and Context-Switching
  12. Annotation Interface
  13. Input Format.
  14. MoE Convergence Monitoring.
  15. User Interface.
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Annotation interface used for the revision of NEL triples.