Autoregressive Direct Preference Optimization

TL;DR

Autoregressive Direct Preference Optimization (ADPO) extends Direct Preference Optimization by incorporating the autoregressive nature of large language models through a prefix-closure domain $\mathcal{Y}^{*}$. This yields a prefix-wise Bradley–Terry model with energies $E_{1}^{*}$ and $E_{2}^{*}$ that align with autoregressive priors, and a loss that moves the summation outside the log-sigmoid, preserving KL-constrained optimality. Theoretical contributions establish that any additively decomposed reward can be represented by an autoregressive policy (Theorem 1) and reveal two independent sequence length measures, token length $\mu$ and feedback length $\mu'$, guiding granularity choices. The framework defines static and adaptive granularity families, enabling token- or prefix-level feedback, and experiments across math reasoning and conversation tasks show ADPO and its contrastive variant cADPO outperform DPO and cDPO, with adaptive fine-grained variants often delivering the best results. Overall, ADPO provides a principled, scalable path to finer-grained preference optimization for autoregressive LLMs with practical improvements on diverse benchmarks.

Abstract

Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences. However, the widespread reliance on the response-level Bradley-Terry (BT) model may limit its full potential, as the reference and learnable models are assumed to be autoregressive only after deriving the objective function. Motivated by this limitation, we revisit the theoretical foundations of DPO and propose a novel formulation that explicitly introduces the autoregressive assumption prior to applying the BT model. By reformulating and extending DPO, we derive a novel variant, termed Autoregressive DPO (ADPO), that explicitly integrates autoregressive modeling into the preference optimization framework. Without violating the theoretical foundations, the derived loss takes an elegant form: it shifts the summation operation in the DPO objective outside the log-sigmoid function. Furthermore, through theoretical analysis of ADPO, we show that there exist two length measures to be considered when designing DPO-based algorithms: the token length $μ$ and the feedback length $μ$'. To the best of our knowledge, we are the first to explicitly distinguish these two measures and analyze their implications for preference optimization in LLMs.

Figures (4)

• Figure 1: Static and adaptive families of ADPO. (a) Token and feedback length measures. Each subsequence $z_{i}$ is defined by a strong composition $\xi$. (b--d) Static families with a fixed window size $k$. (e--g) Adaptive families with a fixed number of subsequences. (e) corresponds to DPO. Blue rectangles indicate regions where summation is applied inside the log-sigmoid function.
• Figure 2: Comparison of training dynamics between DPO and ADPO. The evolution of log probabilities for preferred sequences (solid line) and dispreferred sequences (dashed line) during training is shown. Top row: static family with $k=1, 2$. Bottom row: adaptive family with $m=16, 256$.
• Figure 3: Training dynamics (conversation task).
• Figure 4: Prefix-wise reward variance and margin.

Theorems & Definitions (5)

• proof
• proof
• proof
• proof
• proof