DALL: Data Labeling via Data Programming and Active Learning Enhanced by Large Language Models

TL;DR

The paper tackles the NLP labeling bottleneck by integrating data programming, active learning, and large language models. It presents DALL, a config-based labeling framework that uses a structured specification to define labeling functions, combines active learning to target informative instances, and leverages LLMs to assist labeling function refinement and label corrections. Through comparative, ablation, and usability studies, DALL achieves comparable or higher accuracy with substantially lower labeling time, and the modules synergistically improve efficiency and labeling quality. The work provides a practical interactive labeling system and open-source implementation, offering a scalable solution for generating high-quality NLP datasets.

Abstract

Deep learning models for natural language processing rely heavily on high-quality labeled datasets. However, existing labeling approaches often struggle to balance label quality with labeling cost. To address this challenge, we propose DALL, a text labeling framework that integrates data programming, active learning, and large language models. DALL introduces a structured specification that allows users and large language models to define labeling functions via configuration, rather than code. Active learning identifies informative instances for review, and the large language model analyzes these instances to help users correct labels and to refine or suggest labeling functions. We implement DALL as an interactive labeling system for text labeling tasks. Comparative, ablation, and usability studies demonstrate DALL's efficiency, the effectiveness of its modules, and its usability.

Figures (12)

• Figure 1: The DALL Framework. Given an unlabeled dataset, users can create labeling functions to annotate it. DALL uses active learning algorithms to select informative samples and provides LLM-powered analysis to assist users in refining their labeling functions. Its goal is to iteratively enhance labeling quality.
• Figure 2: The labeling function specification. A labeling function consists of two components: task definition and function configuration. The task definition specifies the task type and the set of label categories used throughout the labeling process. The function configuration assigns specific logic to each function that guides how labels are generated.
• Figure 3: An example labeling function in our specification operates in the following steps: (1) Tag spans with span sets. (2) Apply heuristic rules to convert span sets into label categories. (3) Execute an aggregation method to determine the label category for this data. (4) Return it as the final label.
• Figure 4: The LLM module supports the labeling process in the following ways: (a) Expand span sets using spans extracted from samples. (b) Recommend labeling functions from generalizable heuristics. (c) Analyze samples and provide label recommendations for user reference.
• Figure 5: The prompts for the three components of the LLM module: sample analysis, span set expansion, and labeling function recommendation. Text in blue indicates variables whose values depend on the current labeling status.