Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics

Yurong Chen, Yu He, Michael I. Jordan, Fan Yao

TL;DR

This work examines how sampling and reference-policy choices shape alignment of large language models under pairwise preference learning. via Identity Preference Optimization (IPO) and Direct Preference Optimization (DPO), it demonstrates that instance-dependent sampling can improve ranking guarantees while on-policy sampling can drive excessive concentration under structured preferences. It introduces iterative deployment dynamics (MRS-IPO) showing that feedback between the learned policy, sampling, and reference updates can cause convergence, oscillations, or entropy collapse, with parallel insights for DPO. Empirical evaluation on real-world preference data corroborates the theory and yields practical guidance for stabilizing iterative preference learning pipelines in practice.

Abstract

Standard methods for aligning large language models with human preferences learn from pairwise comparisons among sampled candidate responses and regularize toward a reference policy. Despite their effectiveness, the effects of sampling and reference choices are poorly understood theoretically. We investigate these effects through Identity Preference Optimization, a widely used preference alignment framework, and show that proper instance-dependent sampling can yield stronger ranking guarantees, while skewed on-policy sampling can induce excessive concentration under structured preferences. We then analyze iterative alignment dynamics in which the learned policy feeds back into future sampling and reference policies, reflecting a common practice of model-generated preference data. We prove that these dynamics can exhibit persistent oscillations or entropy collapse for certain parameter choices, and characterize regimes that guarantee stability. Our theoretical insights extend to Direct Preference Optimization, indicating the phenomena we captured are common to a broader class of preference-alignment methods. Experiments on real-world preference data validate our findings.

How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics

TL;DR

This work examines how sampling and reference-policy choices shape alignment of large language models under pairwise preference learning. via Identity Preference Optimization (IPO) and Direct Preference Optimization (DPO), it demonstrates that instance-dependent sampling can improve ranking guarantees while on-policy sampling can drive excessive concentration under structured preferences. It introduces iterative deployment dynamics (MRS-IPO) showing that feedback between the learned policy, sampling, and reference updates can cause convergence, oscillations, or entropy collapse, with parallel insights for DPO. Empirical evaluation on real-world preference data corroborates the theory and yields practical guidance for stabilizing iterative preference learning pipelines in practice.

Abstract

Standard methods for aligning large language models with human preferences learn from pairwise comparisons among sampled candidate responses and regularize toward a reference policy. Despite their effectiveness, the effects of sampling and reference choices are poorly understood theoretically. We investigate these effects through Identity Preference Optimization, a widely used preference alignment framework, and show that proper instance-dependent sampling can yield stronger ranking guarantees, while skewed on-policy sampling can induce excessive concentration under structured preferences. We then analyze iterative alignment dynamics in which the learned policy feeds back into future sampling and reference policies, reflecting a common practice of model-generated preference data. We prove that these dynamics can exhibit persistent oscillations or entropy collapse for certain parameter choices, and characterize regimes that guarantee stability. Our theoretical insights extend to Direct Preference Optimization, indicating the phenomena we captured are common to a broader class of preference-alignment methods. Experiments on real-world preference data validate our findings.
Paper Structure (67 sections, 37 theorems, 237 equations, 28 figures)

This paper contains 67 sections, 37 theorems, 237 equations, 28 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. IPO as a KL-regularized preference objective
  5. Sampling Shapes IPO Optima
  6. Axiom of (im)possibility under sampling
  7. Skewed sampling amplifies skewness
  8. Long-Term Effects of Iterative IPO
  9. Condorcet cycles can induce policy oscillation
  10. Strong transitivity can lead to entropy collapse
  11. Experiments
  12. Experimental setup and system dynamics
  13. Illustrative examples on parameter sensitivity
  14. Aggregated results
  15. Conclusions
  16. ...and 52 more sections

Key Result

Proposition 0

Given preference matrix $P\in\mathcal{P}$, the IPO problem with sampling strategy $\bm{\mu}\in\Delta_K^\circ$ and reference policy $\bm{\pi}_{\mathrm{ref}}\in\Delta_K^\circ$ is equivalent to solving the convex optimization problem which admits a unique closed-form solution $\bm{\pi}^{\star}$: where $\odot$ denotes element-wise multiplication.

Figures (28)

  • Figure 1: Policy evolution of MRS-IPO on $P$ with a cyclic structure in the first $100$ iterations. Compared to the baseline (left), increasing either $\alpha$ (center) or $\beta\lambda$ (right) induces oscillations.
  • Figure 2: Policy evolution of MRS-IPO on a ST preference matrix $P$ in the first 30 iterations. Compared to the baseline (left), increasing either $\alpha$ (center) or $\beta\lambda$ (right) induces more extreme policies.
  • Figure 3: Mean $\pm$ standard deviation of time-averaged variance of $\{\pi_t\}^T_{t=1}$ across all cyclic matrices as $(\alpha,\beta\lambda)$ varies.
  • Figure 4: Mean $\pm$ standard deviation of the final policy entropy $H(\bm{\pi}_T)$ across all ST matrices $(\alpha,\beta\lambda)$ varies.
  • Figure 5: Effect of varying $\alpha$ under cyclic preferences, with $\beta = 2.0$ and $\lambda = 0.8$.
  • ...and 23 more figures

Theorems & Definitions (70)

  • Proposition 0
  • Definition 1
  • Definition 2
  • Proposition 2
  • Theorem 3
  • Proposition 3
  • Definition 4
  • Theorem 5
  • Definition 6
  • Theorem 7
  • ...and 60 more