Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models

TL;DR

<3-5 sentence high-level summary>Boundary ambiguity from large option spaces and inherent token/position biases hinder LLM-based text classification. The authors propose a two-stage framework that first reduces the candidate set via self-reduction (ITR and CBWR) and then applies pairwise comparisons with a Contrastive CoT (PC-CoT) reasoning to refine the decision boundary. Across Banking77, HWU64, LIU54, and Clinc150, and among 11 LLMs, the approach yields substantial gains and reduces bias, demonstrating robustness to option count and arrangement. The method shows strong generalization across model types and tasks, supported by public code for reproducibility and further research.

Abstract

Text classification is a crucial task encountered frequently in practical scenarios, yet it is still under-explored in the era of large language models (LLMs). This study shows that LLMs are vulnerable to changes in the number and arrangement of options in text classification. Our extensive empirical analyses reveal that the key bottleneck arises from ambiguous decision boundaries and inherent biases towards specific tokens and positions. To mitigate these issues, we make the first attempt and propose a novel two-stage classification framework for LLMs. Our approach is grounded in the empirical observation that pairwise comparisons can effectively alleviate boundary ambiguity and inherent bias. Specifically, we begin with a self-reduction technique to efficiently narrow down numerous options, which contributes to reduced decision space and a faster comparison process. Subsequently, pairwise contrastive comparisons are employed in a chain-of-thought manner to draw out nuances and distinguish confusable options, thus refining the ambiguous decision boundary. Extensive experiments on four datasets (Banking77, HWU64, LIU54, and Clinic150) verify the effectiveness of our framework. Furthermore, benefitting from our framework, various LLMs can achieve consistent improvements. Our code and data are available in \url{https://github.com/Chuge0335/PC-CoT}.

Figures (10)

• Figure 1: LLMs in text classification suffer from two main challenges: (a) ambiguous decision boundary, which arises from the explosion of options and their similarity. (b) inherent bias, favoring specific optional positions or tokens. These challenges lead to decreased classification accuracy.
• Figure 2: The classification accuracy decreases across various LLMs as the similarity among options increases. "L" denotes the LLaMA-2-Chat series, while "Q" represents the Qwen-Chat series.
• Figure 3: The influence of option number and arrangement on LLM performance. Full results are in the Appendix \ref{['sec:0']}.
• Figure 4: An overview of our framework for text classification with numerous options. It comprises two main stages: (1) Self Reduction (CBWR) which eliminates extraneous options, and (2) Contrastive CoT Comparison (PC-CoT) that conducts pairwise comparison. In CBWR, distinct option "windows" are induced via clustering to avoid confusion due to similarity. For pairwise comparison, our PC-CoT employs a contrastive chain-of-thought technique to highlight similarities and differences among the paired options for final decision making.
• Figure 5: The confusion matrix visualization illustrates the debiasing effect of our framework on LLaMA70B. In the original few-shot setting, the model showed a preference for "movies" and "convert", while in the ZS-CoT setting, it preferred "iot/hue_lightchange" and "recommendation/movies". After applying our framework, the preference bias clearly shifted.