Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Adaptation of Reinforcement Learning Agents to Sudden Environmental Change

Jonathan Clifford Balloch

TL;DR

This work develops a formal framework and practical methods for online test-time adaptation (OTTA) in reinforcement learning. It demonstrates that efficient adaptation hinges on two pillars: exploration strategies that prioritize diverse, stochastic, task-agnostic data and knowledge-preservation techniques that selectively update reusable prior information. The thesis introduces NovGrid to study novelty in sequential decision tasks, and proposes three core solutions: (i) Dual Objectivity Priority Sampling (DOPS) to align data with learning objectives in model-based RL; (ii) WorldCloner, a neurosymbolic world-model approach that preserves prior knowledge via a symbolic rule model and enables imagination-based policy adaptation; and (iii) Concept Bottleneck World Models (CBWMs) to ground latent representations in interpretable concepts, aiding knowledge preservation and adaptation. Together, these contributions advance OTTA theory and provide practical pathways for deploying adaptive RL systems in non-stationary real-world environments, with demonstrated gains in sample efficiency, adaptation speed, and interpretability. The work also highlights the trade-offs between neural and symbolic representations for rapid adaptation and sets the stage for future research on extended novelty definitions, safe exploration, and scalable symbolic grounding in RL.

Abstract

Real-world autonomous decision-making systems, from robots to recommendation engines, must operate in environments that change over time. While deep reinforcement learning (RL) has shown an impressive ability to learn optimal policies in stationary environments, most methods are data intensive and assume a world that does not change between training and test time. As a result, conventional RL methods struggle to adapt when conditions change. This poses a fundamental challenge: how can RL agents efficiently adapt their behavior when encountering novel environmental changes during deployment without catastrophically forgetting useful prior knowledge? This dissertation demonstrates that efficient online adaptation requires two key capabilities: (1) prioritized exploration and sampling strategies that help identify and learn from relevant experiences, and (2) selective preservation of prior knowledge through structured representations that can be updated without disruption to reusable components.

Efficient Adaptation of Reinforcement Learning Agents to Sudden Environmental Change

TL;DR

This work develops a formal framework and practical methods for online test-time adaptation (OTTA) in reinforcement learning. It demonstrates that efficient adaptation hinges on two pillars: exploration strategies that prioritize diverse, stochastic, task-agnostic data and knowledge-preservation techniques that selectively update reusable prior information. The thesis introduces NovGrid to study novelty in sequential decision tasks, and proposes three core solutions: (i) Dual Objectivity Priority Sampling (DOPS) to align data with learning objectives in model-based RL; (ii) WorldCloner, a neurosymbolic world-model approach that preserves prior knowledge via a symbolic rule model and enables imagination-based policy adaptation; and (iii) Concept Bottleneck World Models (CBWMs) to ground latent representations in interpretable concepts, aiding knowledge preservation and adaptation. Together, these contributions advance OTTA theory and provide practical pathways for deploying adaptive RL systems in non-stationary real-world environments, with demonstrated gains in sample efficiency, adaptation speed, and interpretability. The work also highlights the trade-offs between neural and symbolic representations for rapid adaptation and sets the stage for future research on extended novelty definitions, safe exploration, and scalable symbolic grounding in RL.

Abstract

Real-world autonomous decision-making systems, from robots to recommendation engines, must operate in environments that change over time. While deep reinforcement learning (RL) has shown an impressive ability to learn optimal policies in stationary environments, most methods are data intensive and assume a world that does not change between training and test time. As a result, conventional RL methods struggle to adapt when conditions change. This poses a fundamental challenge: how can RL agents efficiently adapt their behavior when encountering novel environmental changes during deployment without catastrophically forgetting useful prior knowledge? This dissertation demonstrates that efficient online adaptation requires two key capabilities: (1) prioritized exploration and sampling strategies that help identify and learn from relevant experiences, and (2) selective preservation of prior knowledge through structured representations that can be updated without disruption to reusable components.
Paper Structure (104 sections, 1 theorem, 32 equations, 29 figures, 14 tables, 4 algorithms)

This paper contains 104 sections, 1 theorem, 32 equations, 29 figures, 14 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Thesis Statement
  3. Outline
  4. Situating the Work
  5. Reinforcement Learning
  6. Fundamentals of Reinforcement Learning
  7. Model-free and Model-based Approaches to Reinforcement Learning
  8. Model-Free Reinforcement Learning
  9. Model-based Reinforcement Learning
  10. Dreamer World Model
  11. Latent Space Actor Critic
  12. Interaction as Sampling in RL
  13. Exploration
  14. Sampling for Learning and Experience Replay
  15. Transfer Learning and Novelty
  16. ...and 89 more sections

Key Result

Theorem A.7.1

Let actor $\pi_{\theta}$ and critic $v_{\psi}$ be randomly initialized models such that their outputs are distributed as $U(s_t)$. If transition predictor $g_\phi^*$ and reward predictor $\mathcal{R}_\phi^*$ are optimized to predict the online task dynamics, then for small $\epsilon$, $\mathcal{L}_0

Figures (29)

  • Figure 1: The agent-environment interaction that is fundamental to reinforcement learning.
  • Figure 2: The NovGrid environments, where the agent (red triangle) must get to the goal (green box). The novelties are not directly observable; the agent must experience the novelty to be aware of it. Top: pre-novelty only a yellow key opens a door; post-novelty only the blue key opens the door. Bottom: pre-novelty the lava gives a -1 reward and is a terminal state; post-novelty the lava is safe to walk on.
  • Figure 3: Evaluation metrics illustrated against a notional performance curve for an agent.
  • Figure 4: Environments and novelties used to evaluate the exploration algorithms and their characteristics, including discrete and continuous control environments.
  • Figure 5: Full learning and adaptation process of eleven RL exploration algorithms on the DoorKeyChange novelty problem from NovGrid balloch2022novgrid. The agents first learn a task assuming a stationary MDP. The rate of learning at this stage is convergence efficiency. At time step 5,000,000 novelty is injected into the environment, transferring from $MDP_\mathrm{source}$ to $MDP_\mathrm{target}$, often causing a performance drop-off. The algorithms then recover their performance as they learn the new world transition dynamics. The rate of learning at this stage is adaptive efficiency. The maximum episode reward is the final adaptive performance, which may not always be as high as pre-novelty performance.
  • ...and 24 more figures

Theorems & Definitions (3)

  • Definition 1
  • Theorem A.7.1
  • proof