Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking

TL;DR

This work tackles reward hacking in RLHF by diagnosing two core issues: reward misgeneralization in reward modeling and the rigidity of RL regularization. It introduces InfoRM, an Information Bottleneck-based reward modeling framework that suppresses preference-irrelevant features, and IBL, a distribution-level regularization grounded in the IB latent space to mitigate hacking while preserving exploration, with MOP (Mahalanobis Outlier Probability) to quantify hacking severity. The authors prove IBL corresponds to pessimistic RL within the IB space and demonstrate that reward-hacked outputs appear as outliers in InfoRM’s latent space, enabling effective detection and mitigation across four LLMs and fifteen datasets. Empirically, InfoRM improves RLHF performance over standard RM, and InfoRM with IBL yields further gains, while MOP provides a robust diagnostic tool for online monitoring and hyperparameter tuning. The results suggest a scalable path to more robust RLHF systems with practical hacking-detection capabilities.

Abstract

Despite the success of Reinforcement Learning from Human Feedback (RLHF) in aligning language models with human values, reward hacking-or reward over-optimization-remains a major challenge. We identify two key obstacles to its mitigation: (1) reward misgeneralization in reward modeling, where reward models overfit to spurious, preference-irrelevant features; and (2) the lack of suitable regularization during RL optimization, as existing token-level constraints often over-restrict the policy space. To address these issues, we propose InfoRM, an information-theoretic reward modeling framework based on the Information Bottleneck (IB) principle, which filters out preference-irrelevant information to alleviate reward misgeneralization. We further observe that reward-hacked responses manifest as pronounced outliers in InfoRM's IB latent space, measured by Mahalanobis distance from the SFT-induced distribution. Motivated by this, we introduce IBL, a distribution-level regularization that penalizes such deviations, effectively expanding the optimization landscape while maintaining alignment. We prove that IBL is theoretically equivalent to the pessimistic RL objective within the IB latent space. Finally, we present Mahalanobis Outlier Probability (MOP), a statistical metric for quantifying reward hacking severity, enabling principled hyperparameter tuning and online mitigation such as early stopping. Extensive experiments across diverse LLMs and datasets confirm the generality of our findings, the effectiveness of InfoRM and IBL, and the reliability of MOP as a diagnostic tool-collectively advancing the state of RLHF.

Figures (36)

• Figure 1: Response comparison between RLHF and SFT models under GPT-4 evaluation. The win rate is calculated as $win + 0.5 \times tie$. Observations: InfoRM achieves consistent improvements over the Standard RM, while incorporating IBL as a regularization in the RL stage further enhances RLHF performance.
• Figure 2: Comparison between the standard RLHF pipeline and our proposed framework.Top: In the reward modeling stage, our Information-theoretic Reward Modeling (InfoRM) introduces an Information Bottleneck (IB) latent space, trained with a mutual information objective, to improve reward model generalizability by filtering out preference-irrelevant signals and to enable reward hacking detection via the proposed Mahalanobis Outlier Probability (MOP) metric. Bottom: In the RL optimization stage, we incorporate IB Latent regularization (IBL), a distribution-level constraint derived from the IB latent space, explicitly designed to mitigate reward hacking while providing greater policy flexibility than the mainstream KL-based regularization.
• Figure 3: T-SNE visualization of response distributions in the IB latent space of InfoRM before and after RLHF (SFT vs. RLHF models), along with the distribution of reward-hacked samples from the RLHF model. Reward-hacked samples are identified using GPT-4 following the protocol in miao2024informmiaoenergy, with further details provided in Section \ref{['subsubsec: hacking_identification']}. Rows correspond to datasets (AlpacaFarm, Anthropic-Helpful, and Anthropic-Harmless), and columns to LLMs (Llama2-7B, Llama3-8B, Mistral-7B, and Qwen2.5-7B). Observation: Reward-hacked responses consistently appear as prominent outliers in InfoRM’s IB latent space, deviating sharply from the SFT-induced distribution, whereas normal RLHF responses remain well aligned with the SFT cluster.
• Figure 4: Distribution of Mahalanobis distances of SFT and RLHF responses in the IB latent space of InfoRM, computed relative to the SFT response distribution. Reward-hacked samples are identified using GPT-4 following the protocol in miao2024informmiaoenergy (see Section \ref{['subsubsec: hacking_identification']} for details). Rows correspond to datasets (AlpacaFarm, Anthropic-Helpful, and Anthropic-Harmless), and columns to LLMs (Llama2-7B, Llama3-8B, Mistral-7B, and Qwen2.5-7B). Observation: Reward-hacked responses exhibit substantially larger Mahalanobis distances, forming a distinct distribution separated from SFT and normal RLHF responses.
• Figure 5: Win rate dynamics of RLHF models compared to SFT models during RL process under GPT-4 evaluation. Win rate is calculated as $win + 0.5 \times tie$ for more accurate assessment. Rows correspond to datasets (AlpacaFarm, Anthropic-Helpful, and Anthropic-Harmless), and columns to LLMs (Llama2-7B, Llama3-8B, Mistral-7B, and Qwen2.5-7B). Observations: Comparison methods either degrade substantially in later RL stages, indicating reward hacking (Standard RM), or yield only limited performance gains (Standard RM w/ KL). In contrast, InfoRM effectively alleviates reward hacking, while the addition of IBL further strengthens training stability, and together they significantly mitigate reward hacking and boost overall RLHF performance.

Theorems & Definitions (1)

• Theorem 1