Precision over Diversity: High-Precision Reward Generalizes to Robust Instruction Following

TL;DR

The paper interrogates the prevailing view that data diversity is essential for robust instruction following under reinforcement learning with verifiable rewards. Through systematic experiments comparing hard-only, soft-only, and mixed constraint datasets, it reveals that high-precision, rule-based rewards from hard constraints yield superior generalization and efficiency, while LLM-based rewards suffer from reward hacking and lower error detection. It demonstrates that training on hard constraints develops a transferable meta-skill for instruction following, evidenced by attention patterns that internalize IF capabilities and transfer to unverifiable soft tasks. To harness this insight, the authors propose HPPT, a data-centric refinement combining learnability filtering and constraint simplification, which delivers substantial performance gains and major training-time reductions while preserving general capabilities. Collectively, the work advocates a paradigm shift toward high-precision rewards as the primary driver of robust IF, with practical implications for scalable, efficient RLVR systems.

Abstract

A central belief in scaling reinforcement learning with verifiable rewards for instruction following (IF) tasks is that, a diverse mixture of verifiable hard and unverifiable soft constraints is essential for generalizing to unseen instructions. In this work, we challenge this prevailing consensus through a systematic empirical investigation. Counter-intuitively, we find that models trained on hard-only constraints consistently outperform those trained on mixed datasets. Extensive experiments reveal that reward precision, rather than constraint diversity, is the primary driver of effective alignment. The LLM judge suffers from a low recall rate in detecting false response, which leads to severe reward hacking, thereby undermining the benefits of diversity. Furthermore, analysis of the attention mechanism reveals that high-precision rewards develop a transferable meta-skill for IF. Motivated by these insights, we propose a simple yet effective data-centric refinement strategy that prioritizes reward precision. Evaluated on five benchmarks, our approach outperforms competitive baselines by 13.4\% in performance while achieving a 58\% reduction in training time, maintaining strong generalization beyond instruction following. Our findings advocate for a paradigm shift: moving away from the indiscriminate pursuit of data diversity toward high-precision rewards.

Figures (8)

• Figure 1: The impact of data distribution (reward precision and constraint diversity) on model generalization performance. The contour lines represent model performance when trained under different data distributions. Counterintuitively, we find that high-precision rewards, even with limited diversity, achieve superior generalization compared to more diverse reward signals.
• Figure 2: The model's training curves in hard-only, soft-only and mixed constraint datasets, respectively. The Soft-only model achieves higher reward scores than the Hard-only model.
• Figure 3: The reward reliability of LLM-as-a-judge under increasing the constraints. It shows a clear degradation in performance emerges with increasing constraint diversity.
• Figure 4: Impact of reward precision with random noise. It shows reward noise is the critical factor of test performance. (a) Under identical noise levels, hard-only and soft-only model yield near performance. (b) Notably, with low reward noise, hard-only model also fail to develop generalization capabilities.
• Figure 5: Impact of constraint diversity with using limited constraints. It can not observe a consistent improvement as the number of constraints grow.