The Hidden Link Between RLHF and Contrastive Learning

TL;DR

This work unifies RLHF and Direct Preference Optimization under a mutual-information lens, showing they function as contrastive learning between chosen and rejected samples. Replacing the DV/MINE MI estimator with a Jensen–Shannon surrogate yields Mutual Information Optimisation (MIO), which provides stable gradients and mitigates the gradient starvation that hampers DV-based methods. The authors provide theoretical analyses and empirical evidence across eight reasoning and mathematical benchmarks, showing MIO often outperforms prior alignment methods and avoids the late-stage chosen-reward collapse. This approach offers a principled, scalable path to enhance reasoning and mathematical capabilities in aligned LLMs while improving training stability.

Abstract

Alignment of large language models (LLMs) with human values has recently garnered significant attention, with prominent examples including the canonical yet costly Reinforcement Learning from Human Feedback (RLHF) and the simple Direct Preference Optimization (DPO). In this work, we demonstrate that both RLHF and DPO can be interpreted from the perspective of mutual information (MI) maximization, uncovering a profound connection to contrastive learning. Within this framework, both RLHF and DPO can be interpreted as methods that performing contrastive learning based on the positive and negative samples derived from base model, leveraging the Donsker-Varadhan (DV) lower bound on MI (equivalently, the MINE estimator). Such paradigm further illuminates why RLHF may not intrinsically incentivize reasoning capacities in LLMs beyond what is already present in the base model. Building on the perspective, we replace the DV/MINE bound with the Jensen-Shannon (JS) MI estimator and propose the Mutual Information Optimization (MIO). Comprehensive theoretical analysis and extensive empirical evaluations demonstrate that MIO mitigates the late-stage decline in chosen-likelihood observed in DPO, achieving competitive or superior performance across various challenging reasoning and mathematical benchmarks.

Figures (5)

• Figure 1: Toy model setupyan20253dpropertiesidentifyingchallengesdpo. Left: the optimal policy where the highlighted blocks represent optimal responses. Right: preference dataset construction.
• Figure 2: Overview of the DPO and MIO dynamics.From left to right, the figures show the initial state and the likelihood dynamics for chosen/rejected/unseen responses in Scenarios 1 to 4, similar to the left diagram in Figure 2: (1) both chosen and rejected responses are very small, (2) chosen is normal and rejected is very small, (3) chosen is small and rejected is normal, and (4) both chosen and rejected is normal.
• Figure 3: Training curves on Mistral-7B-SFT. Unlike DPO, IPO, and ORPO, MIO increases the rewards of chosen responses while suppressing rejected ones, thereby avoiding the "synchronous collapse" observed in prior methods.
• Figure 4: Left: Estimated mutual information under the bivariate Gaussian setting with varying $\rho$, using MINE and JSD estimators with $K=1$ negative sample. While both methods track the ground-truth trend, JSD is consistently than MINE. Right: Gradient variance of each estimator. The JSD estimator yields substantially lower variance, indicating more stable optimization behavior under limited negative sampling.
• Figure 5: MIO leverages the reward signal more effectively than NCA-Pair.

Theorems & Definitions (6)

• Proposition 3.1: Selective suppression of negatives
• Proposition 3.2: Self-regulating positive gradient
• Definition A.1
• Definition B.1
• Theorem H.1: Gradient starvation for DV/MINE
• Theorem H.2: Linear decay near zero under Lipschitz critics