Repairing Reward Functions with Human Feedback to Mitigate Reward Hacking
Stephane Hatgis-Kessell, Logan Mondal Bhamidipaty, Emma Brunskill
TL;DR
The paper tackles reward misalignment and reward hacking by proposing Preference-Based Reward Repair (PBRR), which repairs a human-specified proxy reward $\hat{r}_{proxy}$ through a transition-dependent correction $g(s,a,s')$, yielding $\hat{r} = \hat{r}_{proxy} + g$. It learns $g$ from human trajectory preferences via a Bradley–Terry model, using a three-term loss that regularizes corrections to only misranked transitions and keeps aligned transitions unchanged, while exploring with a reference policy. Theoretical results show sublinear regret in both known and unknown dynamics, and empirical results across pandemic, glucose, traffic, and AI-safety domains demonstrate data-efficient repair and robust improvement over baselines that learn rewards from scratch or repair via alternative methods. The findings suggest that data-efficient reward alignment can be achieved by targeted exploration combined with a principled correction objective, reducing reliance on large human preference datasets. The work has practical significance for deploying RL in high-stakes settings where human objectives are only partially observable, and it provides a reusable framework for aligning agent behavior with human intent using limited feedback.
Abstract
Human-designed reward functions for reinforcement learning (RL) agents are frequently misaligned with the humans' true, unobservable objectives, and thus act only as proxies. Optimizing for a misspecified proxy reward function often induces reward hacking, resulting in a policy misaligned with the human's true objectives. An alternative is to perform RL from human feedback, which involves learning a reward function from scratch by collecting human preferences over pairs of trajectories. However, building such datasets is costly. To address the limitations of both approaches, we propose Preference-Based Reward Repair (PBRR): an automated iterative framework that repairs a human-specified proxy reward function by learning an additive, transition-dependent correction term from preferences. A manually specified reward function can yield policies that are highly suboptimal under the ground-truth objective, yet corrections on only a few transitions may suffice to recover optimal performance. To identify and correct for those transitions, PBRR uses a targeted exploration strategy and a new preference-learning objective. We prove in tabular domains PBRR has a cumulative regret that matches, up to constants, that of prior preference-based RL methods. In addition, on a suite of reward-hacking benchmarks, PBRR consistently outperforms baselines that learn a reward function from scratch from preferences or modify the proxy reward function using other approaches, requiring substantially fewer preferences to learn high performing policies.