Active Learning Methods for Efficient Data Utilization and Model Performance Enhancement

TL;DR

This paper addresses the annotation bottleneck in data-driven ML by surveying Active Learning (AL) methods. It surveys uncertainty-based sampling, model- and data-centric strategies, Bayesian and interactive approaches, and connections to domain adaptation and transfer learning. Key contributions include synthesis of methodologies, benchmarks, evaluations, and real-world applications, along with a discussion of challenges and future directions. The work provides guidance for researchers and practitioners on choosing acquisition strategies, measurement criteria, and evaluation practices to achieve data-efficient, high-performance models.

Abstract

In the era of data-driven intelligence, the paradox of data abundance and annotation scarcity has emerged as a critical bottleneck in the advancement of machine learning. This paper gives a detailed overview of Active Learning (AL), which is a strategy in machine learning that helps models achieve better performance using fewer labeled examples. It introduces the basic concepts of AL and discusses how it is used in various fields such as computer vision, natural language processing, transfer learning, and real-world applications. The paper focuses on important research topics such as uncertainty estimation, handling of class imbalance, domain adaptation, fairness, and the creation of strong evaluation metrics and benchmarks. It also shows that learning methods inspired by humans and guided by questions can improve data efficiency and help models learn more effectively. In addition, this paper talks about current challenges in the field, including the need to rebuild trust, ensure reproducibility, and deal with inconsistent methodologies. It points out that AL often gives better results than passive learning, especially when good evaluation measures are used. This work aims to be useful for both researchers and practitioners by providing key insights and proposing directions for future progress in active learning.

Figures (8)

• Figure 1: Structure of this Active Learning (AL) survey.
• Figure 2: The TiDAL framework builds on the insight that the training dynamics (TD) of samples can vary, even when their final predicted probabilities $p(y^*|x)$ are identical (upper row). This motivates the use of intermediate training signals by leveraging TD as a source of rich information. To address the computational burden of tracking TD across large-scale unlabeled data, the framework employs a prediction module to estimate TD, rather than explicitly recording it for every sample (lower row).
• Figure 3: Illustration of general cutting-plane method on single iteration.
• Figure 4: Illustration of active learning procedure implemented by ALBench 2207.13339 based on YMIR huang2021ymir. Green indicates labeled data, yellow indicates mined data, and red indicates raw data.
• Figure 5: Desai et al. [1908.02454] proposed framework integrating weak supervision into the active learning pipeline. It features an adaptive supervision module that dynamically escalates the level of supervision, enabling the transition to stronger supervision modes when necessary during model training.