Utility-Learning Tension in Self-Modifying Agents
Charles L. Wang, Keir Dorchen, Peter Jin
TL;DR
The paper identifies a sharp learnability boundary for self-modifying agents: distribution-free PAC guarantees are preserved only when the policy-reachable hypothesis family has uniformly bounded capacity, unifying representational, architectural, algorithmic, metacognitive, and substrate edits under a single capacity criterion. It introduces a computable Two-Gate guardrail (validation margin $\tau$ and capacity cap $K(m)$) that enforces monotone risk improvement and VC-bounded generalization at VC rates. The authors show that allowing unbounded capacity growth through self-modifications can destroy learnability, while capacity-aware policies maintain safety and provide oracle inequalities. They also provide a comprehensive framework linking axis reductions, URP/Local-URP concepts, and proxy-cap bounds, with extensive proofs and experiments validating the theory. Practically, this work guides sustainable open-ended self-improvement by constraining capacity growth rather than suppressing self-modification altogether, offering a principled path toward safe, long-horizon autonomous learning systems.
Abstract
As systems trend toward superintelligence, a natural modeling premise is that agents can self-improve along every facet of their own design. We formalize this with a five-axis decomposition and a decision layer, separating incentives from learning behavior and analyzing axes in isolation. Our central result identifies and introduces a sharp utility--learning tension, the structural conflict in self-modifying systems whereby utility-driven changes that improve immediate or expected performance can also erode the statistical preconditions for reliable learning and generalization. Our findings show that distribution-free guarantees are preserved iff the policy-reachable model family is uniformly capacity-bounded; when capacity can grow without limit, utility-rational self-changes can render learnable tasks unlearnable. Under standard assumptions common in practice, these axes reduce to the same capacity criterion, yielding a single boundary for safe self-modification. Numerical experiments across several axes validate the theory by comparing destructive utility policies against our proposed two-gate policies that preserve learnability.