Mitigating Length Bias in RLHF through a Causal Lens

TL;DR

The paper tackles length bias in RLHF reward models by casting the problem in a causal framework and introducing counterfactual data augmentation to disentangle content quality from verbosity. It constructs two classes of counterfactuals—length-fixed/content-fixed—to isolate content effects from length, enabling training signals that reflect semantic quality rather than surface length. Empirically, the approach reduces length-driven reward bias, improves length-controlled accuracy, and yields policy outputs that are shorter and more informative without sacrificing overall alignment performance. The work demonstrates that causal reasoning and controlled counterfactuals can substantially enhance reward modeling in RLHF, with practical implications for producing concise, content-rich model outputs in real-world applications.

Abstract

Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF-trained reward models often exhibit length bias -- a systematic tendency to favor longer responses by conflating verbosity with quality. We propose a causal framework for analyzing and mitigating length bias in RLHF reward modeling. Central to our approach is a counterfactual data augmentation method that generates response pairs designed to isolate content quality from verbosity. These counterfactual examples are then used to train the reward model, enabling it to assess responses based on content quality independently of verbosity. Specifically, we construct (1) length-divergent pairs with similar content and (2) content-divergent pairs of similar length. Empirical evaluations show that our method reduces length bias in reward assignment and leads to more concise, content-focused outputs from the policy model. These findings demonstrate that the proposed approach effectively reduces length bias and improves the robustness and content sensitivity of reward modeling in RLHF pipelines.

Figures (9)

• Figure 1: Comparison of original perspective and our perspective on reward modeling process.
• Figure 2: Response dimension over an observation edit ($P_1$) and a counterfactual ($P_2$).
• Figure 3: Overview of our method. We generate length- and content-fixed counterfactuals and use them for bias diagnosis and reward model training.
• Figure 4: Experimental pipeline with edge styles indicating flow type. Purple nodes represent data transformation stages; orange nodes algorithmic processing steps. Black arrows data transformation; red dashed arrows model-based processing or state change.
• Figure 5: Reward distribution across response lengths on RewardBench-1 (top) and RewardBench-2 (bottom).