Pushing The Limit of LLM Capacity for Text Classification

TL;DR

This paper investigates whether specialized LLMs can meaningfully advance text classification beyond traditional PLMs and prompting approaches. It introduces RGPT, an adaptive boosting framework that constructs multiple strong base learners by adaptively weighting training samples and fine-tuning LLMs, then recurrently ensembles them via prompts that incorporate prior predictions and error rates. The authors demonstrate, across four benchmarks, that RGPT achieves state-of-the-art results and can even surpass average human performance in a zero-shot/few-shot setting, while being broadly compatible with different base models. The contribution shows that boosting LLM capacity through iterative refinement and recurrent ensembling is a viable path to robust, less-prompt-sensitive text classification, with significant implications for practical deployment and future research.

Abstract

The value of text classification's future research has encountered challenges and uncertainties, due to the extraordinary efficacy demonstrated by large language models (LLMs) across numerous downstream NLP tasks. In this era of open-ended language modeling, where task boundaries are gradually fading, an urgent question emerges: have we made significant advances in text classification under the full benefit of LLMs? To answer this question, we propose RGPT, an adaptive boosting framework tailored to produce a specialized text classification LLM by recurrently ensembling a pool of strong base learners. The base learners are constructed by adaptively adjusting the distribution of training samples and iteratively fine-tuning LLMs with them. Such base learners are then ensembled to be a specialized text classification LLM, by recurrently incorporating the historical predictions from the previous learners. Through a comprehensive empirical comparison, we show that RGPT significantly outperforms 8 SOTA PLMs and 7 SOTA LLMs on four benchmarks by 1.36% on average. Further evaluation experiments show a clear surpassing of RGPT over human classification.

Figures (6)

• Figure 1: Overview of RGPT.
• Figure 2: Recurrent ensembling $K$ base learners.
• Figure 3: Performance of RGPT with increasing number of learners.
• Figure 4: The training loss of RGPT.
• Figure 5: Distribution of training samples and initial test samples during K iterations.