Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Observational Overfitting in Reinforcement Learning

Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, Behnam Neyshabur

TL;DR

The paper investigates observational overfitting in model-free RL, where agents latch onto observation features that correlate with rewards but are irrelevant to underlying dynamics. It introduces a general (f,g)-scheme to create distributions over POMDPs by varying only the observation function while keeping the base MDP fixed. Through theory and experiments on convex 1-step LQR, projected Gym environments, deconvolutional projections, and CoinRun, the authors show that implicit regularization arises with overparameterization and architecture choice, influencing generalization beyond traditional SL bounds. The work highlights that standard RL generalization theories and margin-based bounds do not fully explain these phenomena and suggests directions for future research on architecture design and broader RL settings. Their framework offers a principled way to study how observation-space design shapes RL generalization and overfitting.

Abstract

A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).

Observational Overfitting in Reinforcement Learning

TL;DR

The paper investigates observational overfitting in model-free RL, where agents latch onto observation features that correlate with rewards but are irrelevant to underlying dynamics. It introduces a general (f,g)-scheme to create distributions over POMDPs by varying only the observation function while keeping the base MDP fixed. Through theory and experiments on convex 1-step LQR, projected Gym environments, deconvolutional projections, and CoinRun, the authors show that implicit regularization arises with overparameterization and architecture choice, influencing generalization beyond traditional SL bounds. The work highlights that standard RL generalization theories and margin-based bounds do not fully explain these phenomena and suggests directions for future research on architecture design and broader RL settings. Their framework offers a principled way to study how observation-space design shapes RL generalization and overfitting.

Abstract

A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).

Paper Structure

This paper contains 28 sections, 4 theorems, 45 equations, 16 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Motivation and Related Work
  3. What Happens in Observation Space?
  4. Notation
  5. Setup
  6. Architecture and Implicit Regularization
  7. Experiments
  8. Overparamterized LQR
  9. Projected Gym Environments
  10. Deconvolutional Projections
  11. Observational Overfitting in CoinRun
  12. Overparametrization
  13. Do State-Action Margin Distributions Predict Generalization in RL?
  14. Conclusion
  15. Full Plots for LQR and fg-Gym
  16. ...and 13 more sections

Key Result

Proposition 1

Suppose that $W_{\theta} \sim \mathrm{Unif}(O(p, n))$ and $W_{c} \sim \mathrm{Unif}(O(n))$. Then

Figures (16)

  • Figure 1: Example of observational overfitting in Sonic. Saliency maps highlight (in red) the top-left timer and background objects because they are correlated with progress.
  • Figure 2: (a) Visual Analogy of the Observation Function. (b) Our combinations for 1-D (top) and 2-D (bottom) images for synthetic tasks.
  • Figure 3: (Left) We show that the generalization gap vs noise dimension is tight as the noise dimension increases, showing that this bound is accurate. (Middle and Right) LQR Generalization Gap vs Number of Intermediate Layers. We plotted different $\Phi = \sum_{i=0}^{j} \frac{\left\lVert A\right\rVert_{*}}{\left\lVert A\right\rVert}$ terms without exponents, as powers of those terms are monotonic transforms since $\frac{\left\lVert A\right\rVert_{*}}{\left\lVert A\right\rVert} \ge 1\> \> \forall A$ and $\left\lVert A\right\rVert_{*} = \left\lVert A\right\rVert_{F}, \left\lVert A\right\rVert_{1}$. We see that the naive spectral bound diverges at 2 layers, and the weight-counting sums are too loose.
  • Figure 4: Each Mujoco task is given 10 training levels (randomly sampling $g_{\theta}$ parameters). We used a 2-layer ReLU policy, with 128 hidden units each. Dimensions of outputs of $(f,g)$ were $(30, 100)$ respectively.
  • Figure 5: Effects of Depth.
  • ...and 11 more figures

Theorems & Definitions (7)

  • Proposition 1
  • proof
  • Theorem 1
  • Proposition 2
  • proof
  • Proposition 3
  • proof