Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling

TL;DR

The paper tackles length bias in RLHF reward modeling and policy optimization by introducing Rc-BT, a Response-conditioned Bradley-Terry framework that explicitly differentiates human semantic preferences from response length constraints. It augments the training data with length-aware instructions and defines two augmented prompts to create paired preferences, enabling the reward model and policy optimization to reason about length without sacrificing semantics. Rc-RM and Rc-DPO extend this approach to reward modeling and direct preference optimization, respectively, and experiments across Qwen and Llama models demonstrate reduced length bias, improved length-instruction following, and better semantic quality compared to baselines and prior methods. The work suggests a practical path to more robust RLHF systems, with demonstrated generalization to different biases and datasets, albeit at increased computational cost and with remaining questions for very large models.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has achieved considerable success in aligning large language models (LLMs) by modeling human preferences with a learnable reward model and employing a reinforcement learning algorithm to maximize the reward model's scores. However, these reward models are susceptible to exploitation through various superficial confounding factors, with length bias emerging as a particularly significant concern. Moreover, while the pronounced impact of length bias on preference modeling suggests that LLMs possess an inherent sensitivity to length perception, our preliminary investigations reveal that fine-tuned LLMs consistently struggle to adhere to explicit length instructions. To address these two limitations, we propose a novel framework wherein the reward model explicitly differentiates between human semantic preferences and response length requirements. Specifically, we introduce a $\textbf{R}$esponse-$\textbf{c}$onditioned $\textbf{B}$radley-$\textbf{T}$erry (Rc-BT) model that enhances the model's capability in length bias mitigating and length instruction following, through training on our augmented dataset. Furthermore, we propose the Rc-RM and Rc-DPO algorithm to leverage the Rc-BT model for reward modeling and direct policy optimization (DPO) of LLMs, simultaneously mitigating length bias and promoting adherence to length instructions. Extensive experiments across various foundational models and datasets demonstrate the effectiveness and generalizability of our approach.

Figures (11)

• Figure 1: The relationships between response lengths and scores of reward models (Baseline) trained with Qwen2-1.5B-Instruct and Llama-3.1-8B-Instruct, evaluated on different evaluation datasets, reveal a severe length bias.
• Figure 2: To illustrate the distinct data formats across different methods, we present the maximum length instruction case: (a) The conventional RLHF ouyang2022training with standard preference pair ($x$, $y_w$, $y_l$) ; (b) LIFT with augmented format ($x_l$, $y_l$, $y_w$); and (c) Our method (Rc-BT) with two preference pairs ($x$, $x_l^1$, $y_w$) and ($x_l^2$, $x$, $y_l$). The term length constraint refers to the maximum allowable length of the response as specified in the length instruction $x_l$, $x_l^1$, or $x_l^2$. The black dashed lines indicate the data utilized by each method.
• Figure 3: Analysis of reward scores across models on $\mathcal{D}_{eval}^{ml}$ as a function of response length, with smaller changes indicating reduced length bias.
• Figure 4: (a). Response rewriting template for the chosen response $y_w^{(i)}$; (b). Response expansion template for the rejected response $y_l^{(i)}$; (c). Response compression template for the rejected response $y_l^{(i)}$; (d). Quality consistency verification template for assessing the quality consistency between the rewritten and the original responses.
• Figure 5: The trajectories of Quality Eval Acc, Length Eval Acc, and Training Loss (Smoothed) for different reward models trained on $\text{LIFT-plus}_2^{reverse}$ across training steps.