Policy-labeled Preference Learning: Is Preference Enough for RLHF?

TL;DR

Policy-labeled Preference Learning (PPL) extends RLHF by modeling human preferences with regret and explicitly labeling the behavior policy to address likelihood mismatch. Grounded in the MaxEnt framework, PPL links $Q^\pi$, $V^\pi$ and the entropy term via $\pi^*(a|s) = \exp(\alpha^{-1}(Q^{\pi^*}(s,a) - V^{\pi^*}(s)))$, and shows that maximizing regret-based objectives is equivalent to minimizing the sequential forward KL divergence to observed behavior. Theoretical contributions include a bijection between reward equivalence classes and alpha-optimal soft Q-functions, a unique fixed point for the soft Bellman operator, and a policy-deviation relation $Q^{\pi^*}_*(s,a) - Q^{\pi}_*(s,a) = \alpha \bar{D}_{KL}(\pi || \pi^*;s,a)$. Empirically, PPL yields strong performance on heterogeneous offline MetaWorld datasets and competitive online results with fewer parameters than reward-based baselines, demonstrating robust policy alignment under data diversity and partial labeling. Overall, the work advances RLHF by incorporating behavior policy information and a regret-based learning objective to stabilize and improve preference-based learning in both offline and online regimes.

Abstract

To design rewards that align with human goals, Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent technique for learning reward functions from human preferences and optimizing policies via reinforcement learning algorithms. However, existing RLHF methods often misinterpret trajectories as being generated by an optimal policy, causing inaccurate likelihood estimation and suboptimal learning. Inspired by Direct Preference Optimization framework which directly learns optimal policy without explicit reward, we propose policy-labeled preference learning (PPL), to resolve likelihood mismatch issues by modeling human preferences with regret, which reflects behavior policy information. We also provide a contrastive KL regularization, derived from regret-based principles, to enhance RLHF in sequential decision making. Experiments in high-dimensional continuous control tasks demonstrate PPL's significant improvements in offline RLHF performance and its effectiveness in online settings.

Figures (19)

• Figure 1: Visualization of 5000 samples in Bin-Picking-v2 environment. While the ground-truth reward (left) is sparse and mainly provided upon task completion, regret (right) is more evenly distributed across all timesteps, making it a more informative score function for partial trajectory evaluation.
• Figure 2: Unlike existing DPO algorithms, PPL aligns segment likelihoods by incorporating behavior policies. It reweights gradients based on closeness to the optimal policy, forming a contrastive learning framework.
• Figure 3: Illustration of the likelihood mismatch problem. Although the behavior policy $\pi$ differs from the optimal policy $\pi^*$, the learning process incorrectly assumes all data is generated by $\pi^*$. As a result, while $\pi^*$ prefers $s_1$, this misinterpretation leads to the incorrect conclusion that $s_2$ is preferred, causing suboptimal learning outcomes.
• Figure 4: Distribution of returns in homogeneous vs heterogeneous offline dataset in Button-Press-v2.
• Figure 5: Ablation on deterministic pseudo-labeling. We compare the average performance of PPL and PPL-deterministic across six environments in MetaWorld. The dashed line indicates the point where BC pretraining stops.

Theorems & Definitions (15)

• Definition 3.1
• Lemma 3.2: Structural Condition for $\alpha$-optimality
• Lemma 3.3: Unique Fixed Point of Soft Bellman $\pi$-operator
• Theorem 3.4: Policy Deviation Theorem
• Corollary 3.5
• Corollary 3.6
• Lemma : \ref{['lemma: 121 correspendence']}
• proof
• Lemma : \ref{['lemma: Bellman pi equation']}
• proof