Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Reinforcement Learning in Non-Stationary Context-Driven Environments

Pouya Hamadanian, Arash Nasr-Esfahany, Malte Schwarzkopf, Siddartha Sen, Mohammad Alizadeh

TL;DR

This paper tackles online reinforcement learning in non-stationary, context-driven environments where the environment dynamics shift with exogenous contexts, causing catastrophic forgetting. It introduces Locally Constrained Policy Optimization (LCPO), an on-policy method that anchors policy outputs on past experiences using out-of-distribution samples to constrain updates for current contexts, thereby reducing forgetting without requiring task labels. LCPO integrates reservoir-buffered old experiences, an OOD difference detector, and a KL-based optimization constraint to bound policy changes, achieving performance close to a prescient offline agent across a suite of Mujoco, classic control, and computer-systems tasks. The approach demonstrates strong robustness to detector thresholds and buffer sizes, and its code is publicly available, making it a practical tool for real-world non-stationary RL deployments.

Abstract

We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to "catastrophic forgetting" (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance. We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic and real context traces, and find that it outperforms a variety of baselines in the non-stationary setting, while achieving results on-par with a "prescient" agent trained offline across all context traces. LCPO's source code is available at https://github.com/pouyahmdn/LCPO.

Online Reinforcement Learning in Non-Stationary Context-Driven Environments

TL;DR

This paper tackles online reinforcement learning in non-stationary, context-driven environments where the environment dynamics shift with exogenous contexts, causing catastrophic forgetting. It introduces Locally Constrained Policy Optimization (LCPO), an on-policy method that anchors policy outputs on past experiences using out-of-distribution samples to constrain updates for current contexts, thereby reducing forgetting without requiring task labels. LCPO integrates reservoir-buffered old experiences, an OOD difference detector, and a KL-based optimization constraint to bound policy changes, achieving performance close to a prescient offline agent across a suite of Mujoco, classic control, and computer-systems tasks. The approach demonstrates strong robustness to detector thresholds and buffer sizes, and its code is publicly available, making it a practical tool for real-world non-stationary RL deployments.

Abstract

We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to "catastrophic forgetting" (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance. We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic and real context traces, and find that it outperforms a variety of baselines in the non-stationary setting, while achieving results on-par with a "prescient" agent trained offline across all context traces. LCPO's source code is available at https://github.com/pouyahmdn/LCPO.
Paper Structure (66 sections, 2 theorems, 18 equations, 9 figures, 10 tables, 3 algorithms)

This paper contains 66 sections, 2 theorems, 18 equations, 9 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation.
  4. Non-stationary contexts.
  5. Goal.
  6. Online .
  7. Related Work
  8. Non-stationary .
  9. Latent Context Inference.
  10. Catastrophic Forgetting.
  11. Constrained Optimization.
  12. Locally-Constrained Policy Optimization
  13. Illustrative Example
  14. .
  15. Key Insight.
  16. ...and 51 more sections

Key Result

Lemma B.1

Given a monotonically decreasing sequence $\{\alpha_i\}_{i=1}^{\infty}$ that satisfies the following conditions: Consider any subsequence $\{\beta_j\}_{j=1}^{\infty}$, where for any $1 \le j$, there exists $N \times (j-1) < i \le N \times j$ such that $\beta_j = \alpha_i$. Prove that:

Figures (9)

  • Figure 1: A 3x3 grid-world problem with two modes and the optimal path visualized in blue. (a) In the 'No Trap' mode, the center square is safe to pass through. (b) In the 'Trap' mode, the agent must avoid the trap with a longer path. (c) Episodic return across time in the grid environment. (d and e) Total variation distance between learned and optimal policy outputs for the (d) 'No Trap' mode, and the (e) 'Trap Active' mode at the blue cell (lower is better). Tabular A2C and LCPO remember the optimal decision for either context during shaded regions and instantly attain optimal returns when the environment switches.
  • Figure 2: A sample context process $z_t$, and detected change-points at two thresholds. Teasing meaningful task boundaries is difficult for this process, but defining an metric is intuitive.
  • Figure 3: CDF of normalized lifelong returns, where 0/1 denote the lowest/highest returns among agents. Shaded regions denote 95% confidence intervals. (a) LCPO outperforms all online agents, and remains the closest to prescient policies. (b) LCPO is affected by the threshold $\sigma$, but still outperforms baselines.
  • Figure 4: CDF of normalized returns of LCPO in gym environments with various buffer sizes. Shaded regions denote 95% confidence intervals. LCPO loses performance with $n_b<500$.
  • Figure 5: Architecture of LCPO.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Lemma B.1
  • proof
  • Theorem B.2
  • proof