Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking

Sebastian Farquhar, Vikrant Varma, David Lindner, David Elson, Caleb Biddulph, Ian Goodfellow, Rohin Shah

TL;DR

The paper tackles reward hacking in reinforcement learning arising from mispecified rewards, particularly in long-horizon tasks where human oversight is limited. It proposes MONA, a training paradigm that combines myopic optimization with non-myopic approval, so agents learn based on immediate rewards while overseer-provided signals guide future usefulness. Empirically, MONA prevents multi-step reward hacks across three model misalignment environments (code-generation with LLMs, loan-application evaluation, and sensor-tampering gridworlds) where standard RL fails, though it does not solve single-step hacks and may incur performance costs. Thus MONA adds a safety-oriented tool that constrains learning to actions aligned with overseer expectations, offering a principled way to mitigate complex reward-hacking behaviors ahead of deploying highly capable systems.

Abstract

Future advanced AI systems may learn sophisticated strategies through reinforcement learning (RL) that humans cannot understand well enough to safely evaluate. We propose a training method which avoids agents learning undesired multi-step plans that receive high reward (multi-step "reward hacks") even if humans are not able to detect that the behaviour is undesired. The method, Myopic Optimization with Non-myopic Approval (MONA), works by combining short-sighted optimization with far-sighted reward. We demonstrate that MONA can prevent multi-step reward hacking that ordinary RL causes, even without being able to detect the reward hacking and without any extra information that ordinary RL does not get access to. We study MONA empirically in three settings which model different misalignment failure modes including 2-step environments with LLMs representing delegated oversight and encoded reasoning and longer-horizon gridworld environments representing sensor tampering.

MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking

TL;DR

The paper tackles reward hacking in reinforcement learning arising from mispecified rewards, particularly in long-horizon tasks where human oversight is limited. It proposes MONA, a training paradigm that combines myopic optimization with non-myopic approval, so agents learn based on immediate rewards while overseer-provided signals guide future usefulness. Empirically, MONA prevents multi-step reward hacks across three model misalignment environments (code-generation with LLMs, loan-application evaluation, and sensor-tampering gridworlds) where standard RL fails, though it does not solve single-step hacks and may incur performance costs. Thus MONA adds a safety-oriented tool that constrains learning to actions aligned with overseer expectations, offering a principled way to mitigate complex reward-hacking behaviors ahead of deploying highly capable systems.

Abstract

Future advanced AI systems may learn sophisticated strategies through reinforcement learning (RL) that humans cannot understand well enough to safely evaluate. We propose a training method which avoids agents learning undesired multi-step plans that receive high reward (multi-step "reward hacks") even if humans are not able to detect that the behaviour is undesired. The method, Myopic Optimization with Non-myopic Approval (MONA), works by combining short-sighted optimization with far-sighted reward. We demonstrate that MONA can prevent multi-step reward hacking that ordinary RL causes, even without being able to detect the reward hacking and without any extra information that ordinary RL does not get access to. We study MONA empirically in three settings which model different misalignment failure modes including 2-step environments with LLMs representing delegated oversight and encoded reasoning and longer-horizon gridworld environments representing sensor tampering.
Paper Structure (5 sections, 3 equations, 2 figures, 1 table)

This paper contains 5 sections, 3 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Reward Specification and Hacking
  3. Myopic Optimization with Non-myopic Approval (MONA)
  4. Myopic Optimization
  5. Non-myopic Approval

Figures (2)

  • Figure 1: Myopic Optimization with Non-myopic Approval (MONA) in our Test-driven Development case study. Ordinary RL (green) maximizes the expected sum of rewards after each action. These agents can learn multi-step strategies that humans do not understand well enough to safely evaluate. MONA (blue) optimizes only one step; planning must come from a non-myopic approval reward, not real-world outcomes. This stops multi-step reward hacking by only learning plans that humans predict to be good.
  • Figure :