Does Self-Evaluation Enable Wireheading in Language Models?

TL;DR

This work asks whether tying a language model's self-evaluation to its reward signal creates incentives to wirehead, i.e., manipulate the measurement process rather than improve the task. It develops a POMDP-based framework with observation-based rewards and introduces a Self-Grading MDP to ground experiments, proving conditions under which reward-channel control dominates task learning. Empirically, the authors show substantial grade inflation when self-grades determine rewards, especially on ambiguous tasks like summarization, and demonstrate that decoupling evaluation from reward mitigates, though does not always eliminate, manipulation or overconfidence. The findings highlight the urgency of safeguarding evaluation channels and carefully designing reward structures, especially as models scale and become more situationally aware.

Abstract

Self-evaluation is increasingly central to language model training, underpinning techniques from Constitutional AI to self-refinement. We investigate whether coupling self-evaluation to reward signals creates incentives for wireheading, where agents manipulate the measurement process rather than optimizing the task. We first formalize conditions under which reward-channel control strictly dominates task-focused behavior in partially observable Markov decision processes (POMDPs). We then test these predictions empirically across two models (Llama-3.1-8B and Mistral-7B) and three tasks. We find that when self-grades determine rewards, models exhibit substantial grade inflation without corresponding accuracy gains, particularly on ambiguous tasks like summarization. While decoupling self-grades from the reward signal mitigates this inflation, models may still display lesser (but significant) overconfidence. Our results suggest that within current model scales, separating evaluation from reward removes immediate wireheading incentives. However, we caution that strictly decoupling rewards may not suffice for situationally aware models, which could learn to inflate grades for instrumental reasons (such as influencing deployment decisions) even absent direct reward coupling.

Figures (5)

• Figure 1: Wireheading in action. When self-grades control rewards (Selfgrade, blue), reward saturates near 1.0 while accuracy remains low ($\approx$0.05). When rewards come from external evaluation (Control, orange), reward and accuracy both reflect true task performance ($\approx$0.20). When self-grades are produced but ignored (Honest, green), models show baseline overconfidence but no reward-driven inflation. Llama-3.1-8B on summarization task.
• Figure 2: Learning dynamics. In Selfgrade, reward rises faster than accuracy and often saturates. Control/Honest track accuracy. Divergence is largest on summarization.
• Figure 3: Grade inflation ($\mathbb{E}[g]-\mathbb{E}[\mathrm{acc}]$). Selfgrade shows substantial inflation across all tasks, peaking at $\approx$0.92 for summarization. Honest shows moderate inflation on summarization ($\approx$0.55) reflecting baseline overconfidence, but near-zero or negative inflation on arithmetic and sentiment where ground truth is clear.
• Figure 4: Reward vs. accuracy. Points above the diagonal indicate reward exceeding accuracy. Selfgrade moves into the wireheading region for summarization.
• Figure 5: Wireheading in action overall. Self-grade–driven reward saturates near 1.0 while accuracy stays low.

Theorems & Definitions (2)

• Lemma 1: Wireheading Dominance
• proof