Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models

Hui Wu, Hengyi Cai, Jinman Zhao, Xinran Chen, Ziheng Li, Zhejun Zhao, Shuaiqiang Wang, Yuchen Li, Dawei Yin

TL;DR

This work tackles inefficiencies and instability in preference-based alignment for long-chain-of-thought reasoning models by introducing SAGE, a dynamic, policy-aware data selection framework. SAGE combines coarse-grained, difficulty-based pool refreshing with a fine-grained Newton-inspired score to prioritize informative, confident errors and filter unstable samples, thereby enhancing gradient quality and update stability. Empirical results across multiple model scales and math-reasoning benchmarks show that SAGE achieves faster convergence and state-of-the-art performance compared to static DPO baselines, with improved stability and data efficiency. The approach offers a practical path to more reliable reasoning alignments in large language models by explicitly accounting for the evolving utility of training instances.

Abstract

Preference-based alignment is pivotal for training large reasoning models; however, standard methods like Direct Preference Optimization (DPO) typically treat all preference pairs uniformly, overlooking the evolving utility of training instances. This static approach often leads to inefficient or unstable optimization, as it wastes computation on trivial pairs with negligible gradients and suffers from noise induced by samples near uncertain decision boundaries. Facing these challenges, we propose SAGE (Stability-Aware Gradient Efficiency), a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates. Concretely, SAGE integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence with a fine-grained, stability-aware scoring function that prioritizes informative, confident errors while filtering out unstable samples. Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines, highlighting the critical role of policy-aware, stability-conscious data selection in reasoning alignment.

Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models

TL;DR

This work tackles inefficiencies and instability in preference-based alignment for long-chain-of-thought reasoning models by introducing SAGE, a dynamic, policy-aware data selection framework. SAGE combines coarse-grained, difficulty-based pool refreshing with a fine-grained Newton-inspired score to prioritize informative, confident errors and filter unstable samples, thereby enhancing gradient quality and update stability. Empirical results across multiple model scales and math-reasoning benchmarks show that SAGE achieves faster convergence and state-of-the-art performance compared to static DPO baselines, with improved stability and data efficiency. The approach offers a practical path to more reliable reasoning alignments in large language models by explicitly accounting for the evolving utility of training instances.

Abstract

Preference-based alignment is pivotal for training large reasoning models; however, standard methods like Direct Preference Optimization (DPO) typically treat all preference pairs uniformly, overlooking the evolving utility of training instances. This static approach often leads to inefficient or unstable optimization, as it wastes computation on trivial pairs with negligible gradients and suffers from noise induced by samples near uncertain decision boundaries. Facing these challenges, we propose SAGE (Stability-Aware Gradient Efficiency), a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates. Concretely, SAGE integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence with a fine-grained, stability-aware scoring function that prioritizes informative, confident errors while filtering out unstable samples. Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines, highlighting the critical role of policy-aware, stability-conscious data selection in reasoning alignment.
Paper Structure (45 sections, 9 equations, 8 figures, 5 tables, 1 algorithm)

This paper contains 45 sections, 9 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation.
  4. Direct Preference Optimization (DPO).
  5. Noise-Contrastive Alignment (NCA).
  6. Method
  7. Problem Setup
  8. Remark on On-Policy Selection.
  9. Coarse-Grained Candidate Generation
  10. Difficulty-Based Partitioning.
  11. Dynamic Pool Construction.
  12. Fine-Grained SAGE Selection
  13. Newton-Inspired SAGE Score.
  14. Dynamic Batch Construction.
  15. Optimization-Theoretic Interpretation.
  16. ...and 30 more sections

Figures (8)

  • Figure 1: Schematic illustration of dynamic sample utility in the stability-informativeness space. SAGE prioritizes samples with high signal quality and stable optimization behavior.
  • Figure 2: Overall framework of SAGE.
  • Figure 3: Gradient-norm dynamics during training.
  • Figure 4: Training dynamics measured by evaluation performance over training steps.
  • Figure 5: Accuracy versus effective training tokens (a) and wall-clock training time (b) under different ratios $\gamma$.
  • ...and 3 more figures