Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nonstationary Reinforcement Learning with Linear Function Approximation

Huozhi Zhou, Jinglin Chen, Lav R. Varshney, Ashish Jagmohan

TL;DR

This work tackles nonstationary reinforcement learning with linear function approximation in episodic MDPs under a drift budget $B$. It introduces LSVI-UCB-Restart, an optimistic least-squares value-iteration method with periodic restarts, and Ada-LSVI-UCB-Restart, a parameter-free variant that adapts to unknown variation budgets. The authors prove a first minimax dynamic regret lower bound for nonstationary linear MDPs and derive near-optimal upper bounds: when local variations are known, $ ilde{O}(B^{1/3}d^{4/3}H^{4/3}T^{2/3})$; when variations are unknown, $ ilde{O}(B^{1/4}d^{5/4}H^{5/4}T^{3/4})$, with Ada-LSVI-UCB-Restart achieving the same rate without knowledge of $B$. Empirical results on synthetic nonstationary linear MDPs demonstrate the effectiveness and computational efficiency of the proposed methods compared to baselines. The work advances principled, scalable RL in nonstationary environments and provides a foundation for adaptive, function-approximation-based learning under drift.

Abstract

We consider reinforcement learning (RL) in episodic Markov decision processes (MDPs) with linear function approximation under drifting environment. Specifically, both the reward and state transition functions can evolve over time but their total variations do not exceed a $\textit{variation budget}$. We first develop $\texttt{LSVI-UCB-Restart}$ algorithm, an optimistic modification of least-squares value iteration with periodic restart, and bound its dynamic regret when variation budgets are known. Then we propose a parameter-free algorithm $\texttt{Ada-LSVI-UCB-Restart}$ that extends to unknown variation budgets. We also derive the first minimax dynamic regret lower bound for nonstationary linear MDPs and as a byproduct establish a minimax regret lower bound for linear MDPs unsolved by Jin et al. (2020). Finally, we provide numerical experiments to demonstrate the effectiveness of our proposed algorithms.

Nonstationary Reinforcement Learning with Linear Function Approximation

TL;DR

This work tackles nonstationary reinforcement learning with linear function approximation in episodic MDPs under a drift budget . It introduces LSVI-UCB-Restart, an optimistic least-squares value-iteration method with periodic restarts, and Ada-LSVI-UCB-Restart, a parameter-free variant that adapts to unknown variation budgets. The authors prove a first minimax dynamic regret lower bound for nonstationary linear MDPs and derive near-optimal upper bounds: when local variations are known, ; when variations are unknown, , with Ada-LSVI-UCB-Restart achieving the same rate without knowledge of . Empirical results on synthetic nonstationary linear MDPs demonstrate the effectiveness and computational efficiency of the proposed methods compared to baselines. The work advances principled, scalable RL in nonstationary environments and provides a foundation for adaptive, function-approximation-based learning under drift.

Abstract

We consider reinforcement learning (RL) in episodic Markov decision processes (MDPs) with linear function approximation under drifting environment. Specifically, both the reward and state transition functions can evolve over time but their total variations do not exceed a . We first develop algorithm, an optimistic modification of least-squares value iteration with periodic restart, and bound its dynamic regret when variation budgets are known. Then we propose a parameter-free algorithm that extends to unknown variation budgets. We also derive the first minimax dynamic regret lower bound for nonstationary linear MDPs and as a byproduct establish a minimax regret lower bound for linear MDPs unsolved by Jin et al. (2020). Finally, we provide numerical experiments to demonstrate the effectiveness of our proposed algorithms.

Paper Structure

This paper contains 28 sections, 26 theorems, 77 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Nonstationary bandits
  4. RL with function approximation
  5. Nonstationary RL
  6. Notation
  7. Preliminaries
  8. Minimax Regret Lower Bound
  9. LSVI-UCB-Restart Algorithm
  10. Algorithm Description
  11. Regret Analysis
  12. Known Local Variations
  13. Unknown Local Variation
  14. Ada-LSVI-UCB-Restart: a Parameter-free Algorithm
  15. Algorithm Description
  16. ...and 13 more sections

Key Result

Theorem 1

For any algorithm, the dynamic regret is at least $\Omega(B^{1/3}d^{2/3}H^{1/3}T^{2/3})$ for one nonstationary homogeneous linear MDP instance, if $d\geq 4$, $T\geq 64(d-3)^2H$.

Figures (3)

  • Figure 1: Comparisons of different methods on cumulative reward under two different environments. The results are averaged over 10 trials and the error bars show the standard deviations. The environment changes abruptly in the left subfigure, whereas the environment changes gradually in the right subfigure.
  • Figure 2: Comparisons of different methods on running time for two different environments. The results are averaged over 10 trials and the error bars show the standard deviations. See Appendix \ref{['sec:hardware']} for details on hardware.
  • Figure 3: Graphical illustration of the hard-to-learn linear MDP instances with deterministic reward.

Theorems & Definitions (49)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Theorem 3
  • Theorem 4
  • Corollary 1
  • Remark 1
  • Remark 2
  • ...and 39 more