Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey

Julia Romberg, Christopher Schröder, Julius Gonsior, Katrin Tomanek, Fredrik Olsson

TL;DR

Confronting the data annotation bottleneck in NLP amid advancing LLMs, this paper reassesses active learning adoption through a large-scale community survey. It collects 52 questions from 144 NLP practitioners across academia and industry, analyzes current AL practices, obstacles, and future trends, and compares them to the 2009 survey. The findings show annotation remains essential, LLMs are now the prevalent backbone, and three adoption barriers—setup complexity, uncertain cost reduction, and tooling—persist. The work offers practical recommendations to reduce setup overhead, improve cost estimation, and enhance tooling, and releases an anonymized dataset for further study.

Abstract

Supervised learning relies on data annotation which usually is time-consuming and therefore expensive. A longstanding strategy to reduce annotation costs is active learning, an iterative process, in which a human annotates only data instances deemed informative by a model. Research in active learning has made considerable progress, especially with the rise of large language models (LLMs). However, we still know little about how these remarkable advances have translated into real-world applications, or contributed to removing key barriers to active learning adoption. To fill in this gap, we conduct an online survey in the NLP community to collect previously intangible insights on current implementation practices, common obstacles in application, and future prospects in active learning. We also reassess the perceived relevance of data annotation and active learning as fundamental assumptions. Our findings show that data annotation is expected to remain important and active learning to stay relevant while benefiting from LLMs. Consistent with a community survey from over 15 years ago, three key challenges yet persist -- setup complexity, uncertain cost reduction, and tooling -- for which we propose alleviation strategies. We publish an anonymized version of the dataset.

Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey

TL;DR

Confronting the data annotation bottleneck in NLP amid advancing LLMs, this paper reassesses active learning adoption through a large-scale community survey. It collects 52 questions from 144 NLP practitioners across academia and industry, analyzes current AL practices, obstacles, and future trends, and compares them to the 2009 survey. The findings show annotation remains essential, LLMs are now the prevalent backbone, and three adoption barriers—setup complexity, uncertain cost reduction, and tooling—persist. The work offers practical recommendations to reduce setup overhead, improve cost estimation, and enhance tooling, and releases an anonymized dataset for further study.

Abstract

Supervised learning relies on data annotation which usually is time-consuming and therefore expensive. A longstanding strategy to reduce annotation costs is active learning, an iterative process, in which a human annotates only data instances deemed informative by a model. Research in active learning has made considerable progress, especially with the rise of large language models (LLMs). However, we still know little about how these remarkable advances have translated into real-world applications, or contributed to removing key barriers to active learning adoption. To fill in this gap, we conduct an online survey in the NLP community to collect previously intangible insights on current implementation practices, common obstacles in application, and future prospects in active learning. We also reassess the perceived relevance of data annotation and active learning as fundamental assumptions. Our findings show that data annotation is expected to remain important and active learning to stay relevant while benefiting from LLMs. Consistent with a community survey from over 15 years ago, three key challenges yet persist -- setup complexity, uncertain cost reduction, and tooling -- for which we propose alleviation strategies. We publish an anonymized version of the dataset.

Paper Structure

This paper contains 55 sections, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Findings
  4. Survey Methodology
  5. Descriptive Research Approach
  6. Questions and Branching
  7. Target Audience and Distribution
  8. Consent, Data Usage, and Privacy
  9. Results
  10. Participant Statistics
  11. Data Annotation in Times of LLMs
  12. Overcoming Data Annotation Challenges
  13. Dissemination of active learning
  14. Alternative methods
  15. Contemporary Active Learning
  16. ...and 40 more sections

Figures (9)

  • Figure 1: Development of AL publication numbers in *CL venues over the past 15 years (total for reference). See Appendix \ref{['sec:app:publication-identification']} for methodological details.
  • Figure 2: This diagram illustrates (a) the names and sequence of questions groups, (b) the logical flow, including branching points and conditions, and (c) possible survey paths, ordered by the number of questions.
  • Figure 3: Exploratory histograms of the participants' work sectors, field, educational background, work experience, and place of residence (cf. \ref{['q:VII.1']}). The x-axis is scaled depending on the smallest and largest percentage value.
  • Figure 4: Respondents' assessment of the relevance of data annotation in context of recent advancements in NLP using a 5-point Likert scale (cf. \ref{['q:I.3']}).
  • Figure 5: Obstacles especially faced in the application of AL (a) and participants' general experience with computational methods for a lack of annotated data (b).
  • ...and 4 more figures