GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets

TL;DR

GenCLS++ addresses the challenge of leveraging LLMs for generative text classification by unifying supervised fine-tuning and reinforcement learning within a single framework. It systematically explores diverse training and inference prompt strategies—including in-context learning variants, category definitions, uncertainty labeling, numeric labels, and perplexity-based decoding—and demonstrates that these combinations yield substantial accuracy gains over naive SFT baselines. Empirically, GenCLS++ achieves an average improvement of about 3.46% across seven datasets (4.00% on public data), with RL after SFT providing additional boosts; importantly, explicit reasoning prompts are not generally beneficial for classification. The findings offer practical guidance for prompt design, initialization, and RL application in generative classifiers, and suggest that shorter, direct outputs often suffice for accurate categorization in LLMs.

Abstract

As a fundamental task in machine learning, text classification plays a crucial role in many areas. With the rapid scaling of Large Language Models (LLMs), particularly through reinforcement learning (RL), there is a growing need for more capable discriminators. Consequently, advances in classification are becoming increasingly vital for enhancing the overall capabilities of LLMs. Traditional discriminative methods map text to labels but overlook LLMs' intrinsic generative strengths. Generative classification addresses this by prompting the model to directly output labels. However, existing studies still rely on simple SFT alone, seldom probing the interplay between training and inference prompts, and no work has systematically leveraged RL for generative text classifiers and unified SFT, RL, and inference-time prompting in one framework. We bridge this gap with GenCLS++, a framework that jointly optimizes SFT and RL while systematically exploring five high-level strategy dimensions-in-context learning variants, category definitions, explicit uncertainty labels, semantically irrelevant numeric labels, and perplexity-based decoding-during both training and inference. After an SFT "policy warm-up," we apply RL with a simple rule-based reward, yielding sizable extra gains. Across seven datasets, GenCLS++ achieves an average accuracy improvement of 3.46% relative to the naive SFT baseline; on public datasets, this improvement rises to 4.00%. Notably, unlike reasoning-intensive tasks that benefit from explicit thinking processes, we find that classification tasks perform better without such reasoning steps. These insights into the role of explicit reasoning provide valuable guidance for future LLM applications.

Figures (6)

• Figure 1: An overview of the GenCLS++ framework. It explores diverse combinations of training and inference strategies for classification tasks and incorporates RL to further enhance performance. We conduct comprehensive experiments on seven datasets, encompassing different languages, varying numbers of categories, and diverse data types.
• Figure 2: Comparison of Different Models: Response Length vs. Reward over Steps
• Figure 3: Comparison of model performance across different RL algorithms.
• Figure 4: Visualization of the improvement achieved by changing the inference prompt strategy. Left: improvement in accuracy. Right: improvement in macro-F1. The x-axis represents the dataset, and the y-axis represents the different training strategies. Improvements are highlighted in red, while decreases are shown in blue.
• Figure 5: Visualization of the improvement achieved by changing the inference prompt strategy from fixed-3-shot to similar-3-shot. Left: improvement in accuracy. Right: improvement in macro-F1. The x-axis represents the dataset, and the y-axis represents the different training strategies. Improvements are highlighted in red, while decreases are shown in blue.