Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization

TL;DR

This work reframes Direct Preference Optimization (DPO) within a Bayesian paradigm by introducing the Differential Information Distribution (DID), the distribution over samples that carry evidence to update a reference policy into a target policy. It shows that DPO’s log-ratio reward is the unique Bradley–Terry form compatible with learning the target policy when preferences encode DID, and it derives a power-law relationship between the DID of competing policies that explains observed training dynamics. Through controlled Energy-Based Model experiments and real-data setups, it demonstrates that the entropy of the DID predicts downstream capabilities: high-entropy DID favors open-ended instruction following, while low-entropy DID aligns with factual QA. Collectively, the DID framework provides theoretical grounding and practical guidance for designing and understanding preference-based alignment, linking reward structure, training dynamics, and capabilities in a principled manner.

Abstract

Direct Preference Optimization (DPO) has been widely used for aligning language models with human preferences in a supervised manner. However, several key questions remain unresolved: the rationale behind its log-ratio reward, how the statistical structure of preference datasets shapes its training dynamics, and how those dynamics impact downstream capabilities. We approach these questions from a Bayesian perspective, interpreting the goal of preference optimization as learning the differential information required to update a reference policy into a target policy. To formalize this view, we introduce the Differential Information Distribution (DID), defined as the distribution over samples that carry the Bayesian evidence required to update policies. We introduce three complementary insights by viewing preference optimization through the DID. First, we find that DPO's log-ratio reward is uniquely justified when preferences encode the Differential Information needed to update a reference policy into the target policy. Second, we discuss how commonly observed training dynamics in DPO, including changes in log-likelihood and policy exploration, stem from a power-law DID relationship. Finally, we analyze how training dynamics influence downstream performance using the entropy of DID, a principled measure of uncertainty in the learned information. We observe that learning high-entropy DID improves open-ended instruction-following, while low-entropy DID benefits knowledge-intensive QA. Taken together, our results show that DPO's reward design, training dynamics, and downstream capabilities all emerge as natural consequences of learning Differential Information, offering both a principled theoretical foundation and practical guidance for preference-based alignment.

Figures (12)

• Figure 1: Left: Optimization using $r=\log \pi$ on preference data satisfying the DID power-law. The Jensen-Shannon Divergence $\mathbb D_\mathrm{JS}[q_{{\pi^*}/{\pi_\mathrm{ref}}}\Vert \pi]$ converges to 0, confirming Theorem \ref{['def:DIDD']} that the preference encodes Differential Information. Right: Comparison of $\mathbb D_\mathrm{JS}[\pi^* \Vert \pi]$ using different objectives on the same data with $\tau=4$. Standard DPO ($r=\log(\pi/\pi_\mathrm{ref})$, purple) uniquely converges to $\pi^*$, consistent with Theorem \ref{['th:opt']}.
• Figure 2: Log-likelihood change during DPO training. When chosen responses $y_w$ are sampled from $\pi_\mathrm{ref}$, the log-likelihood of rejected responses $y_\ell$ decreases relative to $\pi_\mathrm{ref}$ (left plot). When rejected samples $y_\ell$ are sampled from $\pi_\mathrm{ref}$, the log-likelihood of chosen responses increases for $\beta\ge1$ (right plot). This confirms the predicted change in log-likelihood of DPO (Theorem \ref{['thm:dyn']}).
• Figure 3: Policy exploration of DPO. Compared to the original dataset $\mathcal{D}$, halving the sampling temperature to form $\mathcal{D}'$ strengthens preferences ($\alpha=2$) and increases the KL-divergence from $\pi_\mathrm{ref}$ under the same KL-penalty $\beta$, consistent with Theorem \ref{['thm:str']}. Increasing the KL-penalty to $2\beta$ when training on $\mathcal{D}'$ restores the divergence to the level obtained with $\mathcal{D}$ using $\beta$, in line with Remark \ref{['remark:beta']}.
• Figure 4: Convergence quality (Jensen-Shannon divergence) between the target $\pi^*$ and the converged policy $\pi$ under varying dataset exponents $\beta_\ell$ and $\beta_w$ (controlling $\pi_\ell$ and $\pi_w$ respectively), and reward scale $\beta_r$. Consistent with Theorem \ref{['thm:opt_rej']}, the best convergence occurs near the diagonal $\beta_\ell=\beta_r$ and $\beta_w=\beta_r$.
• Figure 5: Illustration of policy reinforcement (left) and smoothing (right). The gray region corresponds to $\mathcal{Y}^-=\{y'\in\mathcal{Y}\mid \pi_\mathrm{ref}(y')\approx0\}$, and the light-blue region $\{\pi<\pi_\mathrm{ref}\}$ corresponds to $\{\tilde{y}\in\mathcal{Y} \mid \pi(\tilde{y})<\pi_\mathrm{ref}(\tilde{y})\}$. This plot serves only as an illustrative example and does not represent the true DID $q_{{\pi^*}/{\pi_\mathrm{ref}}}$.

Theorems & Definitions (41)

• Theorem 2.1: Preference vs. Distribution Matching dumoulin2024densityestimationperspectivelearning
• Definition 2.2: Differential Information Distribution
• Theorem 2.3: Likelihood Ratio Representation of Differential Information Distribution
• Theorem 3.1: Preferences Encoding Differential Information
• Theorem 3.2: Optimal Reward for Learning Differential Information
• Corollary 3.2.1: DID Power-Law of DPO
• Theorem 4.1: Log-Likelihood Change of DPO
• Theorem 4.2: Adaptive Policy Exploration of DPO
• Remark 1
• Remark 2