Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tape: A Cellular Automata Benchmark for Evaluating Rule-Shift Generalization in Reinforcement Learning

Enze Pan

TL;DR

Tape introduces a cellular-automata based reinforcement-learning benchmark that isolates out-of-distribution generalization under latent rule shifts by keeping observation and action interfaces fixed while varying the latent update rule. The authors propose standardized OOD protocols and rigorous statistical reporting, supported by a theory appendix that links information gain to conditional mutual information and posterior KL divergence. Empirically, strong in-distribution planning performance does not guarantee robustness under rule shifts, with model-based planners often deteriorating under OOD while task-inference methods like PEARL show relatively stronger resilience, albeit with high variance needing replication. The work argues for combining explicit task inference, robust model-usage control, and strict replication to reliably assess OOD robustness, and provides a practical roadmap for comprehensive validation of RL generalization methods.

Abstract

We present Tape, a controlled reinforcement-learning benchmark designed to isolate out-of-distribution (OOD) failure under latent rule shifts.Tape is derived from one-dimensional cellular automata, enabling precise train/test splits where observation and action spaces are held fixed while transition rules change. Using a reproducible evaluation pipeline, we compare model-free baselines, model-based planning with learned world models, and task-inference (meta-RL) methods. A consistent pattern emerges: methods that are strong in-distribution (ID) can collapse under heldout-rule OOD, and high-variance OOD evaluation can make rankings unstable unless experiments are sufficiently replicated.We provide (i) standardized OOD protocols, (ii) statistical reporting requirements (seeds, confidence intervals, and hypothesis tests), and (iii) information-theoretic identities connecting entropy reduction to conditional mutual information and expected posterior KL divergence, clarifying what "uncertainty reduction" objectives can and cannot guarantee under rule shifts.

Tape: A Cellular Automata Benchmark for Evaluating Rule-Shift Generalization in Reinforcement Learning

TL;DR

Tape introduces a cellular-automata based reinforcement-learning benchmark that isolates out-of-distribution generalization under latent rule shifts by keeping observation and action interfaces fixed while varying the latent update rule. The authors propose standardized OOD protocols and rigorous statistical reporting, supported by a theory appendix that links information gain to conditional mutual information and posterior KL divergence. Empirically, strong in-distribution planning performance does not guarantee robustness under rule shifts, with model-based planners often deteriorating under OOD while task-inference methods like PEARL show relatively stronger resilience, albeit with high variance needing replication. The work argues for combining explicit task inference, robust model-usage control, and strict replication to reliably assess OOD robustness, and provides a practical roadmap for comprehensive validation of RL generalization methods.

Abstract

We present Tape, a controlled reinforcement-learning benchmark designed to isolate out-of-distribution (OOD) failure under latent rule shifts.Tape is derived from one-dimensional cellular automata, enabling precise train/test splits where observation and action spaces are held fixed while transition rules change. Using a reproducible evaluation pipeline, we compare model-free baselines, model-based planning with learned world models, and task-inference (meta-RL) methods. A consistent pattern emerges: methods that are strong in-distribution (ID) can collapse under heldout-rule OOD, and high-variance OOD evaluation can make rankings unstable unless experiments are sufficiently replicated.We provide (i) standardized OOD protocols, (ii) statistical reporting requirements (seeds, confidence intervals, and hypothesis tests), and (iii) information-theoretic identities connecting entropy reduction to conditional mutual information and expected posterior KL divergence, clarifying what "uncertainty reduction" objectives can and cannot guarantee under rule shifts.
Paper Structure (41 sections, 2 theorems, 8 equations, 4 tables)

This paper contains 41 sections, 2 theorems, 8 equations, 4 tables.

Table of Contents

  1. Introduction
  2. Positioning.
  3. A brief inspiration note (optional).
  4. Contributions.
  5. Related Work
  6. Generalization and OOD evaluation in RL.
  7. Model-based RL and world models.
  8. Task inference and meta-RL.
  9. Uncertainty and information gain.
  10. Data augmentation and domain randomization.
  11. The TAPE Benchmark
  12. Task family.
  13. Why CA for rule-shift OOD.
  14. Protocols.
  15. Environment specification.
  16. ...and 26 more sections

Key Result

Theorem C.1

Theorems & Definitions (4)

  • Theorem C.1: Information gain equals conditional mutual information
  • proof
  • Theorem C.2: Information gain equals expected posterior KL
  • proof