On Goodhart's law, with an application to value alignment

TL;DR

This work formalizes Goodhart's law through the model $M=G+\xi$ and analyzes how the top-$\alpha$ correlation $\rho_\alpha$ between the goal $G$ and the proxy $M$ behaves as $\alpha\to0$, showing that the tail of the discrepancy $\xi$ is the key determinant. It proves precise asymptotics across several distributions (exponential, normal, and power-law) and distinguishes weak, strong, and infinite Goodhart by comparing the tails of $G$ and $\xi$, with normal noise yielding a very gradual decay of correlation and thicker-tailed discrepancies enabling negative or even unbounded harm when optimizing $M$. The paper also discusses worst-case scenarios where dependence between $G$ and $\xi$ drives $\mathbb{E}_\alpha[G]$ to extremes, and presents implications for alignment and policy in systems with feedback loops, including illustrative algorithm-user interactions. Overall, the results highlight the critical need for tail-aware, robust metrics and caution against over-reliance on proxies in large-scale automated decision-making contexts.

Abstract

``When a measure becomes a target, it ceases to be a good measure'', this adage is known as {\it Goodhart's law}. In this paper, we investigate formally this law and prove that it critically depends on the tail distribution of the discrepancy between the true goal and the measure that is optimized. Discrepancies with long-tail distributions favor a Goodhart's law, that is, the optimization of the measure can have a counter-productive effect on the goal. We provide a formal setting to assess Goodhart's law by studying the asymptotic behavior of the correlation between the goal and the measure, as the measure is optimized. Moreover, we introduce a distinction between a {\it weak} Goodhart's law, when over-optimizing the metric is useless for the true goal, and a {\it strong} Goodhart's law, when over-optimizing the metric is harmful for the true goal. A distinction which we prove to depend on the tail distribution. We stress the implications of this result to large-scale decision making and policies that are (and have to be) based on metrics, and propose numerous research directions to better assess the safety of such policies in general, and to the particularly concerning case where these policies are automated with algorithms.

Figures (11)

• Figure 1: As $\alpha$ decreases, the measure threshold $m_\alpha$ increases. At some point, the expected goal $\mathbb E_\alpha[G]$ reaches a plateaus. The figure depictes expected goal as a function of the measure threhold (on the left) and as a function of $\log_2(1/\alpha)$ (on the right). The parameters were set at $\varepsilon = 1/256$.
• Figure 2: As $\alpha$ decreases, the measure threshold $m_\alpha$ increases. The expected goal $\mathbb E_\alpha[G]$ increases steadily. The figure depicts expected goal as a function of the measure threshold (on the left) and as a function of $\log_2(1/\alpha)$ (on the right). The parameters were set at $\varepsilon = 1/256$.
• Figure 3: For a uniform goal and a power law noise, as $\alpha$ decreases, the measure threshold $m_\alpha$ increases. The expected value of the goal increases, but then decreases sharply. The figure illustrates this phenomenon as a function of the measure threshold (on the left) and as a function of $\log_2(1/\alpha)$ (on the right). The parameters were set at $\varepsilon = 1/256$ and $\beta = 3.5$.
• Figure 4: Consider a uniform goal and a power law noise. Initially near 1, the correlation between $M$ and $G$ decreases when $\alpha$ decreases, as a function of the measure threshold $m_\alpha$ (on the left) and of $\log_2(1/\alpha)$ (on the right). It even becomes negative, before reaching zero eventually. The plot depicts the case where $\varepsilon = 1/256$ and $\beta=3.5$.
• Figure 5: As $\alpha$ decreases, the measure threshold $m_\alpha$ increases. The expected goal $\mathbb E_\alpha[G]$ increases steadily. The figure depicts the expected goal as a function of the measure threshold (on the left) and as a function of $\log_2(1/\alpha)$ (on the right). The parameters were set at $\beta=3.5$, $\gamma=7$ and $\varepsilon = 1/256$.

Theorems & Definitions (80)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Theorem 4
• proof
• Theorem 5
• proof