Applying LLMs to Active Learning: Towards Cost-Efficient Cross-Task Text Classification without Manually Labeled Data

TL;DR

The paper tackles the high cost of manually labeling data for text classification by introducing an LLM-driven active learning framework that uses GPT as an oracle guided by structured prompts and RoBERTa embeddings to enable cross-task classification without human annotations. It systematically evaluates multiple query strategies within an active learning loop and demonstrates that the approach delivers high accuracy across sentiment, news, and toxic-comment tasks while achieving substantial cost and time savings relative to direct GPT classification. The findings show GPT labeling can match human labeling performance in this setup, making it a viable alternative for large-scale annotation, with active learning providing additional gains over random sampling. Overall, the work highlights the practical potential of combining LLMs with active learning to scale cross-task NLP applications efficiently and suggests avenues for multilingual and broader-domain extensions.

Abstract

Machine learning-based classifiers have been used for text classification, such as sentiment analysis, news classification, and toxic comment classification. However, supervised machine learning models often require large amounts of labeled data for training, and manual annotation is both labor-intensive and requires domain-specific knowledge, leading to relatively high annotation costs. To address this issue, we propose an approach that integrates large language models (LLMs) into an active learning framework, achieving high cross-task text classification performance without the need for any manually labeled data. Furthermore, compared to directly applying GPT for classification tasks, our approach retains over 93% of its classification performance while requiring only approximately 6% of the computational time and monetary cost, effectively balancing performance and resource efficiency. These findings provide new insights into the efficient utilization of LLMs and active learning algorithms in text classification tasks, paving the way for their broader application.

Figures (11)

• Figure 1: Overall Workflow: LLM-driven Active Learning Loop
• Figure 2: Length Distribution of IMDB Dataset
• Figure 3: Length Distribution of AGnews Dataset
• Figure 4: Length Distribution of Jigsaw Toxic Comment Classification Dataset
• Figure 5: Label Distributions