Goodhart's Law in Reinforcement Learning
Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, Joar Skalse
TL;DR
This work investigates how reinforcement learning agents can behave suboptimally with respect to the true objective when trained on imperfect proxy rewards, through the lens of Goodhart's law. It introduces a geometric, occupancy-based framework that recasts policy optimization as a linear program over a convex polytope of occupancy measures and defines a projected reward distance via arg(R0, R1) and a Normalised Drop Height metric to quantify Goodharting. The authors establish a mechanistic explanation for Goodharting in RL, illustrate its ubiquity across diverse environments, and propose two provably robust policy optimization strategies, including an optimal stopping rule with regret guarantees, to avoid the pitfall. They validate these ideas experimentally, showing that early stopping can prevent Goodharting across many setups, albeit at potential cost to true objective performance, and discuss practical considerations for estimating key quantities and extending the framework to reward refinement and broader failure modes.
Abstract
Implementing a reward function that perfectly captures a complex task in the real world is impractical. As a result, it is often appropriate to think of the reward function as a proxy for the true objective rather than as its definition. We study this phenomenon through the lens of Goodhart's law, which predicts that increasing optimisation of an imperfect proxy beyond some critical point decreases performance on the true objective. First, we propose a way to quantify the magnitude of this effect and show empirically that optimising an imperfect proxy reward often leads to the behaviour predicted by Goodhart's law for a wide range of environments and reward functions. We then provide a geometric explanation for why Goodhart's law occurs in Markov decision processes. We use these theoretical insights to propose an optimal early stopping method that provably avoids the aforementioned pitfall and derive theoretical regret bounds for this method. Moreover, we derive a training method that maximises worst-case reward, for the setting where there is uncertainty about the true reward function. Finally, we evaluate our early stopping method experimentally. Our results support a foundation for a theoretically-principled study of reinforcement learning under reward misspecification.