Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Take Goodhart Seriously: Principled Limit on General-Purpose AI Optimization

Antoine Maier, Aude Maier, Tom David

TL;DR

The paper argues that the common Objective Satisfaction Assumption (OSA)—that training yields models that truly optimize the specified objective—fails in realistic settings due to approximation, estimation, and optimization errors, as well as unavoidable misspecification of human intent. Using a learning-paradigm-agnostic framework, it decomposes excess risk into three nonzero components and shows how finite capacity, finite data, and imperfect optimization guarantee deviations from the intended objective; it also highlights performativity and outer vs inner misalignment as entrenched challenges. By connecting these gaps to Goodhart's law, the authors show that under strong optimization pressure proxy targets can diverge from true goals, potentially causing loss of control in General-Purpose AI (GPAI) systems. The work argues for principled limits on GPAI optimization—such as halting criteria or capacity constraints—since the location of the Goodhart breakpoint is intractable to locate in advance, and thus proactive safeguards are essential for safety and robustness.

Abstract

A common but rarely examined assumption in machine learning is that training yields models that actually satisfy their specified objective function. We call this the Objective Satisfaction Assumption (OSA). Although deviations from OSA are acknowledged, their implications are overlooked. We argue, in a learning-paradigm-agnostic framework, that OSA fails in realistic conditions: approximation, estimation, and optimization errors guarantee systematic deviations from the intended objective, regardless of the quality of its specification. Beyond these technical limitations, perfectly capturing and translating the developer's intent, such as alignment with human preferences, into a formal objective is practically impossible, making misspecification inevitable. Building on recent mathematical results, absent a mathematical characterization of these gaps, they are indistinguishable from those that collapse into Goodhart's law failure modes under strong optimization pressure. Because the Goodhart breaking point cannot be located ex ante, a principled limit on the optimization of General-Purpose AI systems is necessary. Absent such a limit, continued optimization is liable to push systems into predictable and irreversible loss of control.

Take Goodhart Seriously: Principled Limit on General-Purpose AI Optimization

TL;DR

The paper argues that the common Objective Satisfaction Assumption (OSA)—that training yields models that truly optimize the specified objective—fails in realistic settings due to approximation, estimation, and optimization errors, as well as unavoidable misspecification of human intent. Using a learning-paradigm-agnostic framework, it decomposes excess risk into three nonzero components and shows how finite capacity, finite data, and imperfect optimization guarantee deviations from the intended objective; it also highlights performativity and outer vs inner misalignment as entrenched challenges. By connecting these gaps to Goodhart's law, the authors show that under strong optimization pressure proxy targets can diverge from true goals, potentially causing loss of control in General-Purpose AI (GPAI) systems. The work argues for principled limits on GPAI optimization—such as halting criteria or capacity constraints—since the location of the Goodhart breakpoint is intractable to locate in advance, and thus proactive safeguards are essential for safety and robustness.

Abstract

A common but rarely examined assumption in machine learning is that training yields models that actually satisfy their specified objective function. We call this the Objective Satisfaction Assumption (OSA). Although deviations from OSA are acknowledged, their implications are overlooked. We argue, in a learning-paradigm-agnostic framework, that OSA fails in realistic conditions: approximation, estimation, and optimization errors guarantee systematic deviations from the intended objective, regardless of the quality of its specification. Beyond these technical limitations, perfectly capturing and translating the developer's intent, such as alignment with human preferences, into a formal objective is practically impossible, making misspecification inevitable. Building on recent mathematical results, absent a mathematical characterization of these gaps, they are indistinguishable from those that collapse into Goodhart's law failure modes under strong optimization pressure. Because the Goodhart breaking point cannot be located ex ante, a principled limit on the optimization of General-Purpose AI systems is necessary. Absent such a limit, continued optimization is liable to push systems into predictable and irreversible loss of control.

Paper Structure

This paper contains 19 sections, 9 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Related Work
  3. Objective Satisfaction Assumption
  4. Learning Framework
  5. Objective function
  6. Hypothesis class
  7. Experience term and population vs empirical distribution
  8. Regularization terms
  9. Why the Objective Satisfaction Assumption Fails
  10. Approximation error
  11. Estimation error
  12. Optimization error
  13. Misgeneralization
  14. Performativity
  15. Misspecification
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Schematic representation of the risks arising from the Objective Satisfaction Assumption (OSA) failure and misspecification. Estimation errors and misspecification systematically introduce a discrepancy between the developer’s true intent and the model’s learned objective. Except in the unlikely case of accidental alignment, strong optimization pressure turns this discrepancy into Goodhart failure modes and ultimately into loss-of-control scenarios. Limiting optimization, e.g., through limits on the optimization algorithm itself or on the model’s representational capacity, can prevent these risks.