Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models

TL;DR

This paper investigates why reinforcement-learning with verifiable rewards yields uneven gains across large language model families, and identifies distributional clarity in probability space as the hidden driver. It introduces the Silhouette Coefficient ($S$) to quantify intra-class compactness and inter-class separation between correct and incorrect response probabilities, and links high $S$ to better RL performance and stable reasoning. A Silhouette-Aware Reweighting strategy, guided by $S$ (and a rectified variant $S'$), is proposed to emphasize low-clarity samples during training, yielding consistent improvements across six mathematical benchmarks and multiple model families, with notable gains on challenging datasets like AIME24 and a strong $r=0.815$ correlation between $S$ and pass rates. The work thereby reframes RL-Friendliness as a trainable structural property of the probability landscape, offering a practical route to enhance RL-based reasoning beyond data-centric approaches.

Abstract

Language model families exhibit striking disparity in their capacity to benefit from reinforcement learning: under identical training, models like Qwen achieve substantial gains, while others like Llama yield limited improvements. Complementing data-centric approaches, we reveal that this disparity reflects a hidden structural property: \textbf{distributional clarity} in probability space. Through a three-stage analysis-from phenomenon to mechanism to interpretation-we uncover that RL-friendly models exhibit intra-class compactness and inter-class separation in their probability assignments to correct vs. incorrect responses. We quantify this clarity using the \textbf{Silhouette Coefficient} ($S$) and demonstrate that (1) high $S$ correlates strongly with RL performance; (2) low $S$ is associated with severe logic errors and reasoning instability. To confirm this property, we introduce a Silhouette-Aware Reweighting strategy that prioritizes low-$S$ samples during training. Experiments across six mathematical benchmarks show consistent improvements across all model families, with gains up to 5.9 points on AIME24. Our work establishes distributional clarity as a fundamental, trainable property underlying RL-Friendliness.

Figures (11)

• Figure 1: Schematic illustration of the Silhouette Coefficient ($S$). We adapt this metric to quantify distributional clarity. High $S$ values represent ideal landscapes with compact and separated clusters, while low values indicate overlapping distributions.
• Figure 2: Per-problem pass rate comparison between Qwen2.5-7B (DAPO) and OctoThinker-8B (DAPO) on AIME 2024. Each point represents a query. Points below the diagonal indicate Qwen achieves higher pass rates. A similar distribution pattern is observed on MATH-500 (see Figure \ref{['fig:pass_rate_comparison_math500']}).
• Figure 3: Probability Distributions: Kernel density estimates of sequence probabilities for correct (top) and incorrect (bottom) responses. Qwen exhibits clear separation, whereas Llama and OctoThinker show significant overlap.
• Figure 4: Impact of distributional structure on pass rates. Queries with positive $S$ values achieve significantly higher performance across all models.
• Figure 5: Error attribution analysis on MATH-500. (a) Proportion of High (Fundamental), Mid (Execution), and Low (Presentation) severity errors across models. (b) Percentage of responses with $S < 0$ (poor distributional clarity) within each error category. High-severity errors are strongly correlated with low distributional clarity.