Bias Fitting to Mitigate Length Bias of Reward Model in RLHF

TL;DR

RLHF relies on reward models that can be gamed by length bias, where longer outputs receive higher rewards regardless of quality. FiMi-RM introduces a three-stage approach: warm-up to establish biased reward modeling, a lightweight fitting model using length encoding and a ResNet to learn a non-linear length–reward relation, and a debiasing stage that decouples length from reward while preserving preference accuracy. The method demonstrates non-linear length bias patterns—strong linearity for short outputs, decelerating growth for mid-lengths, and noise for very long outputs—and delivers more balanced length–reward distributions and improved length-controlled performance across BoN and DPO. These results enhance robust RLHF alignment by mitigating spurious length correlations, with practical implications for controlling verbosity without sacrificing task performance. FiMi-RM thus offers a principled, data-efficient path to reduce reward hacking due to length bias in large language models.

Abstract

Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences. However, RLHF often suffers from reward hacking, wherein policy learning exploits flaws in the trained reward model to maximize reward scores without genuinely aligning with human preferences. A significant example of such reward hacking is length bias, where reward models usually favor longer responses irrespective of actual response quality. Previous works on length bias have notable limitations, these approaches either mitigate bias without characterizing the bias form, or simply assume a linear length-reward relation. To accurately model the intricate nature of length bias and facilitate more effective bias mitigation, we propose FiMi-RM (Bias Fitting to Mitigate Length Bias of Reward Model in RLHF), a framework that autonomously learns and corrects underlying bias patterns. Our approach consists of three stages: First, we train a standard reward model which inherently contains length bias. Next, we deploy a lightweight fitting model to explicitly capture the non-linear relation between length and reward. Finally, we incorporate this learned relation into the reward model to debias. Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution. Furthermore, when applied to alignment algorithms, our debiased reward model improves length-controlled win rate and reduces verbosity without compromising its performance.

Figures (5)

• Figure 1: The overview of our method (part a). First, we use traditional reward model training to initially establish the model's length bias. Second, We employ a lightweight fitting model to fit the reward hacking: given the length of a response, we minimize two losses to make the output of the fitting model as close as possible to that of the reward model. The final step involves debiasing the length in the reward model based on the relation fitted by the fitting model. In addition, we also present the detailed architecture of the $model_f$ (part b).
• Figure 2: Scatter plot of reward versus length, with binned averages (black lines). Our method demonstrates a more balanced reward distribution compared to others, indicating effective debiasing.
• Figure 3: The pairwise comparison of the distribution of responses selected by BoN. The figure indicate that, relative to vanilla RM, our approach demonstrates a stronger inclination toward shorter responses in BoN selection. Although ODIN also mitigates bias toward overly lengthy outputs, it mainly shifts preferences toward medium-length responses rather than enhancing the selection of shorter ones.
• Figure 4: Length distribution differences of chosen and rejected responses in the labeling stage of DPO. The gap between chosen and rejected response length is obversely smaller for our method when comparing to both ODIN and vanilla RM.
• Figure 5: The fitted curve of $model_{f}$ at different steps in training. In general, the curve gradually aligns with the trend of the scatter points and eventually matches their pattern closely. From the last subfigure, the relation exhibits strong linearity at shorter length, with the hacked reward increasing at a nearly constant rate as length grows. However, in the mid-length range, the growth rate begins to taper off. For sequences longer than 200, the correlation becomes increasingly ambiguous.