Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learning

Tongxi Wang, Zhuoyang Xia, Xinran Chen, Shan Liu

TL;DR

This work tackles non-stationary reinforcement learning by tying exploration strength to online measures of environmental drift. It introduces Adaptive Entropy Scheduling (AES), a plug-in mechanism that schedules the entropy coefficient online via drift proxies, coupling the step size to entropy to yield a per-round trade-off between tracking a drifting optimum and maintaining stability. The theory unifies online convex optimization with non-stationary soft-RL, deriving regret bounds that scale with the drift budget and showing how to compute fully-online schedules from observable proxies. Empirically, AES improves robustness and recovery speed across multiple RL carriers (SAC, PPO, SQL, MEow) and task suites, with negligible overhead and stable performance in steady environments. The results advocate a principled, variation-aware approach to exploration that can be integrated broadly into deep RL pipelines.

Abstract

Real-world reinforcement learning often faces environment drift, but most existing methods rely on static entropy coefficients/target entropy, causing over-exploration during stable periods and under-exploration after drift (thus slow recovery), and leaving unanswered the principled question of how exploration intensity should scale with drift magnitude. We prove that entropy scheduling under non-stationarity can be reduced to a one-dimensional, round-by-round trade-off, faster tracking of the optimal solution after drift vs. avoiding gratuitous randomness when the environment is stable, so exploration strength can be driven by measurable online drift signals. Building on this, we propose AES (Adaptive Entropy Scheduling), which adaptively adjusts the entropy coefficient/temperature online using observable drift proxies during training, requiring almost no structural changes and incurring minimal overhead. Across 4 algorithm variants, 12 tasks, and 4 drift modes, AES significantly reduces the fraction of performance degradation caused by drift and accelerates recovery after abrupt changes.

Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learning

TL;DR

This work tackles non-stationary reinforcement learning by tying exploration strength to online measures of environmental drift. It introduces Adaptive Entropy Scheduling (AES), a plug-in mechanism that schedules the entropy coefficient online via drift proxies, coupling the step size to entropy to yield a per-round trade-off between tracking a drifting optimum and maintaining stability. The theory unifies online convex optimization with non-stationary soft-RL, deriving regret bounds that scale with the drift budget and showing how to compute fully-online schedules from observable proxies. Empirically, AES improves robustness and recovery speed across multiple RL carriers (SAC, PPO, SQL, MEow) and task suites, with negligible overhead and stable performance in steady environments. The results advocate a principled, variation-aware approach to exploration that can be integrated broadly into deep RL pipelines.

Abstract

Real-world reinforcement learning often faces environment drift, but most existing methods rely on static entropy coefficients/target entropy, causing over-exploration during stable periods and under-exploration after drift (thus slow recovery), and leaving unanswered the principled question of how exploration intensity should scale with drift magnitude. We prove that entropy scheduling under non-stationarity can be reduced to a one-dimensional, round-by-round trade-off, faster tracking of the optimal solution after drift vs. avoiding gratuitous randomness when the environment is stable, so exploration strength can be driven by measurable online drift signals. Building on this, we propose AES (Adaptive Entropy Scheduling), which adaptively adjusts the entropy coefficient/temperature online using observable drift proxies during training, requiring almost no structural changes and incurring minimal overhead. Across 4 algorithm variants, 12 tasks, and 4 drift modes, AES significantly reduces the fraction of performance degradation caused by drift and accelerates recovery after abrupt changes.
Paper Structure (137 sections, 24 theorems, 259 equations, 1 figure, 8 tables)

This paper contains 137 sections, 24 theorems, 259 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Variation-Aware Non-Stationary RL
  4. Exploration Control: Entropy Scheduling vs. Heuristic Adaptation
  5. Theory
  6. Dynamic mirror descent yields a per-round $\lambda$-trade-off
  7. A fully-online schedule from an observable drift proxy
  8. Bridge to non-stationary maximum-entropy RL
  9. Experiment-Theory Interface.
  10. Experiments
  11. Experiment Setup
  12. Evaluation Metrics
  13. Overall Performance Across Non-stationarity Patterns
  14. Abrupt-change Recovery Time
  15. Summary
  16. ...and 122 more sections

Key Result

Lemma 3.1

Let $\{u_t\}_{t=1}^T\subset\Pi$ be any comparator sequence and set $u_0=u_1$. Assume bounded mirror gradients: there exists $G_\Psi<\infty$ such that and assume $\eta_t$ is nondecreasing. Then the iterates generated by eq:md-update satisfy

Figures (1)

  • Figure 1: Intuition for adaptive exploration in non-stationary environments. The left panel illustrates an environment whose conditions evolve over time (e.g., changes in road conditions, friction, or task objectives). The right panels (a–c) contrast agent behaviors under different exploration regimes: (a) when abrupt environmental changes occur, insufficient exploration leads to delayed adaptation and poor recovery; (b) during stable periods, overly strong exploration introduces unnecessary randomness and degrades performance; (c) by adaptively adjusting exploration strength based on online drift signals, the agent can rapidly recover after changes while maintaining efficient exploitation in stable phases. This figure summarizes the core idea of the paper: treating exploration strength as a one-dimensional control variable driven by environmental drift, balancing fast tracking of changes against avoiding gratuitous randomness.

Theorems & Definitions (37)

  • Lemma 3.1: Dynamic MD inequality
  • Theorem 3.2: $\lambda$-trade-off dynamic regret bound
  • Theorem 3.3: Per-round oracle optimal $\lambda_t$
  • Theorem 3.4: Online AES scheduling driven by an observable proxy
  • Theorem 3.5: AES-RL-Plan: non-stationary soft-RL dynamic regret (principal term)
  • Lemma 2.1: Dynamic MD inequality
  • Theorem 2.2: $\lambda$-trade-off dynamic regret bound
  • Theorem 2.3: Per-round oracle optimal $\lambda_t$
  • Theorem 2.4: Online AES scheduling driven by an observable proxy
  • Theorem 2.5: AES-RL-Plan: non-stationary soft-RL dynamic regret (principal term)
  • ...and 27 more