Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks

Philipp Kohl, Yoka Krämer, Claudia Fohry, Bodo Kraft

TL;DR

The study addresses the annotation burden in named entity recognition by surveying model-agnostic active learning strategies and their evaluation environments. Using a PRISMA-ScR-guided scoping review of Scopus and ACM sources, it maps 106 AL strategies across 62 papers and 57 corpora, revealing a dominance of exploitation-based, uncertainty-driven methods and a reliance on the F1-score for evaluation. It highlights gaps in reporting hardware and timing, and uneven public availability of corpora, proposing a standardized ALE-style evaluation framework to enable fair benchmarking and cross-domain comparisons. The findings provide a practical roadmap for researchers to conduct rigorous empirical comparisons and for practitioners to select appropriate AL strategies in ER settings.

Abstract

We conducted a scoping review for active learning in the domain of natural language processing (NLP), which we summarize in accordance with the PRISMA-ScR guidelines as follows: Objective: Identify active learning strategies that were proposed for entity recognition and their evaluation environments (datasets, metrics, hardware, execution time). Design: We used Scopus and ACM as our search engines. We compared the results with two literature surveys to assess the search quality. We included peer-reviewed English publications introducing or comparing active learning strategies for entity recognition. Results: We analyzed 62 relevant papers and identified 106 active learning strategies. We grouped them into three categories: exploitation-based (60x), exploration-based (14x), and hybrid strategies (32x). We found that all studies used the F1-score as an evaluation metric. Information about hardware (6x) and execution time (13x) was only occasionally included. The 62 papers used 57 different datasets to evaluate their respective strategies. Most datasets contained newspaper articles or biomedical/medical data. Our analysis revealed that 26 out of 57 datasets are publicly accessible. Conclusion: Numerous active learning strategies have been identified, along with significant open questions that still need to be addressed. Researchers and practitioners face difficulties when making data-driven decisions about which active learning strategy to adopt. Conducting comprehensive empirical comparisons using the evaluation environment proposed in this study could help establish best practices in the domain.

Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks

TL;DR

The study addresses the annotation burden in named entity recognition by surveying model-agnostic active learning strategies and their evaluation environments. Using a PRISMA-ScR-guided scoping review of Scopus and ACM sources, it maps 106 AL strategies across 62 papers and 57 corpora, revealing a dominance of exploitation-based, uncertainty-driven methods and a reliance on the F1-score for evaluation. It highlights gaps in reporting hardware and timing, and uneven public availability of corpora, proposing a standardized ALE-style evaluation framework to enable fair benchmarking and cross-domain comparisons. The findings provide a practical roadmap for researchers to conduct rigorous empirical comparisons and for practitioners to select appropriate AL strategies in ER settings.

Abstract

We conducted a scoping review for active learning in the domain of natural language processing (NLP), which we summarize in accordance with the PRISMA-ScR guidelines as follows: Objective: Identify active learning strategies that were proposed for entity recognition and their evaluation environments (datasets, metrics, hardware, execution time). Design: We used Scopus and ACM as our search engines. We compared the results with two literature surveys to assess the search quality. We included peer-reviewed English publications introducing or comparing active learning strategies for entity recognition. Results: We analyzed 62 relevant papers and identified 106 active learning strategies. We grouped them into three categories: exploitation-based (60x), exploration-based (14x), and hybrid strategies (32x). We found that all studies used the F1-score as an evaluation metric. Information about hardware (6x) and execution time (13x) was only occasionally included. The 62 papers used 57 different datasets to evaluate their respective strategies. Most datasets contained newspaper articles or biomedical/medical data. Our analysis revealed that 26 out of 57 datasets are publicly accessible. Conclusion: Numerous active learning strategies have been identified, along with significant open questions that still need to be addressed. Researchers and practitioners face difficulties when making data-driven decisions about which active learning strategy to adopt. Conducting comprehensive empirical comparisons using the evaluation environment proposed in this study could help establish best practices in the domain.
Paper Structure (17 sections, 4 figures, 3 tables)

This paper contains 17 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Definitions
  3. Entity Recognition
  4. Active Learning
  5. Methodology
  6. Search Engines
  7. Review Process
  8. Eligibility Criteria
  9. Results
  10. Active Learning Strategies for ER (Review Question 1)
  11. Corpora (Review Question 2a)
  12. Metrics, Hardware, and Execution Times (Review Questions 2b, 2c)
  13. Research Gaps
  14. Evaluation Environment
  15. Related Work
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Flow diagram for the pool-based active learning cycle following settles_active_2009kohl_ale_2023.
  • Figure 2: Our review process followed the procedure proposed by triccoPRISMAExtensionScoping2018. It is divided into five stages, described in more detail in \ref{['sec:review-process']}. The number of exclusion reasons listed for stage 2) to 4) does not always add up to the total number of excluded records because multiple exclusion criteria can exclude the same record. See the GitHub repository for a detailed list.
  • Figure 3: Publication year of the 2000er for the 62 papers analyzed within this scoping review.
  • Figure 4: The figure shows how many times corpora from specific domains are used in our 62 papers, grouped by the corpus licensing. The majority of the papers use open access corpora for their experiments.