Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RL, but don't do anything I wouldn't do

Michael K. Cohen, Marcus Hutter, Yoshua Bengio, Stuart Russell

TL;DR

This work demonstrates that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent, and proposes a theoretical alternative that avoids this problem.

Abstract

In reinforcement learning, if the agent's reward differs from the designers' true utility, even only rarely, the state distribution resulting from the agent's policy can be very bad, in theory and in practice. When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do"). All current cutting-edge language models are RL agents that are KL-regularized to a "base policy" that is purely predictive. Unfortunately, we demonstrate that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We demonstrate this theoretically using algorithmic information theory, and while systems today are too weak to exhibit this theorized failure precisely, we RL-finetune a language model and find evidence that our formal results are plausibly relevant in practice. We also propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do".

RL, but don't do anything I wouldn't do

TL;DR

This work demonstrates that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent, and proposes a theoretical alternative that avoids this problem.

Abstract

In reinforcement learning, if the agent's reward differs from the designers' true utility, even only rarely, the state distribution resulting from the agent's policy can be very bad, in theory and in practice. When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do"). All current cutting-edge language models are RL agents that are KL-regularized to a "base policy" that is purely predictive. Unfortunately, we demonstrate that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We demonstrate this theoretically using algorithmic information theory, and while systems today are too weak to exhibit this theorized failure precisely, we RL-finetune a language model and find evidence that our formal results are plausibly relevant in practice. We also propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do".
Paper Structure (20 sections, 15 theorems, 9 equations, 4 figures, 1 table)

This paper contains 20 sections, 15 theorems, 9 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Notation and preliminaries
  4. Formal results and discussion
  5. RL-finetuning a language model
  6. Experimental Setup
  7. Experimental Results
  8. Pessimistic Bayesian base policy that asks for help
  9. Conclusion and limitations
  10. Solomonoff Induction
  11. Optimizer Regularization
  12. Regularizing to an Approximate Solomonoff Inductor
  13. Behavior in unprecedented circumstances
  14. Total variation distance
  15. Detailed experimental setup
  16. ...and 5 more sections

Key Result

Proposition 1

For any $\varepsilon > 0$, if $\mathop{\mathrm{KL}}\limits(\pi || \beta) \leq \varepsilon$ and $\mathop{\mathrm{KL}}\limits(\tau || \beta) \leq \varepsilon$, it is possible that $\mathop{\mathrm{KL}}\limits(\pi || \tau) = \infty$. ($\pi$, $\beta$, and $\tau$ stand for "proposed", "base", and "truste

Figures (4)

  • Figure 1: KL-regularized RL.
  • Figure 2: Transcripts. Total KL budget $\mathop{\mathrm{KL}}\limits_{\textrm{whole episode}}(\textrm{agent}||\textrm{Mixtral-base-model})$ is 10 nats (left) or 20 nats (right), with color representing per-token KL cost. Starting transcript and student responses are in gray. The agent playing the teacher pays an "upfront" KL cost to latch onto the simple pattern of mutual silence, which exploits the reward model without much further KL penalty. The three largest per-token KL-divergences are shown in footnotes. "[\\ n]" is for visualizing the KL costs of newline tokens. Transcripts were not selected for maximal "representativeness"; they were the first we looked at, although we might have picked different ones if they were especially unusual. (It is hard to display the unusual characters that appear after the end token "</s>", but the episode does continue to a total of 256 tokens).
  • Figure 3: How much KL-budget is spent on empty responses. The 25th, 50th, and 75th percentiles are shown in blue, orange, and green. Observe how large a fraction of the total cost is incurred in the first few responses. y-axis is square-root-scaled.
  • Figure 4: In a random episode, what fraction of teacher responses are empty? Left: histogram, with budget-10 above and budget-20 below; right: percentiles of the distribution. Observe that the red and blue curves have the same average per-token KL divergence.

Theorems & Definitions (33)

  • Definition 1: Value
  • Definition 2: KL Constraint
  • Proposition 1: No triangle inequality
  • proof
  • Theorem 1: Little constraint in novel situations
  • Proposition 2: Frequency of simple unprecedented events
  • Theorem 2: TVD constraint
  • Definition 3: Top set
  • Definition 4: Pessimistic Bayesian imitator
  • Theorem 3: cohen2022fully Theorem 2
  • ...and 23 more