Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs

Han Zhong, Zhongren Chen, Zhuoran Yang, Zhaoran Wang, Csaba Szepesvári

TL;DR

This work addresses episodic RL under non-stationary dynamics where rewards and transitions drift within budgets Δ_T and Δ_P in a linear kernel MDP. It introduces PROPO, a periodically restarted optimistic policy optimization framework that combines sliding-window policy evaluation with KL-regularized policy updates, plus SW-LSVI-UCB, a sliding-window optimistic value-iteration method. The authors establish a minimax lower bound and show sublinear dynamic regret for PROPO and SW-LSVI-UCB, with near-optimal dependence on the problem dimensions and drift budgets; they also provide parameter-free variants (B-PROPO, B-SW-LSVI-UCB) and validate the theory through synthetic experiments. The results demonstrate that policy optimization can be provably efficient in non-stationary environments and highlight when value-based methods may be preferable under milder non-stationarity. Overall, this work advances the understanding of provable, practical policy-optimization approaches for non-stationary RL with function approximation.

Abstract

We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs). In this setting, both the reward function and the transition kernel are linear with respect to the given feature maps and are allowed to vary over time, as long as their respective parameter variations do not exceed certain variation budgets. We propose the \underline{p}eriodically \underline{r}estarted \underline{o}ptimistic \underline{p}olicy \underline{o}ptimization algorithm (PROPO), which is an optimistic policy optimization algorithm with linear function approximation. PROPO features two mechanisms: sliding-window-based policy evaluation and periodic-restart-based policy improvement, which are tailored for policy optimization in a non-stationary environment. In addition, only utilizing the technique of sliding window, we propose a value-iteration algorithm. We establish dynamic upper bounds for the proposed methods and a minimax lower bound which shows the (near-) optimality of the proposed methods. To our best knowledge, PROPO is the first provably efficient policy optimization algorithm that handles non-stationarity.

Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs

TL;DR

This work addresses episodic RL under non-stationary dynamics where rewards and transitions drift within budgets Δ_T and Δ_P in a linear kernel MDP. It introduces PROPO, a periodically restarted optimistic policy optimization framework that combines sliding-window policy evaluation with KL-regularized policy updates, plus SW-LSVI-UCB, a sliding-window optimistic value-iteration method. The authors establish a minimax lower bound and show sublinear dynamic regret for PROPO and SW-LSVI-UCB, with near-optimal dependence on the problem dimensions and drift budgets; they also provide parameter-free variants (B-PROPO, B-SW-LSVI-UCB) and validate the theory through synthetic experiments. The results demonstrate that policy optimization can be provably efficient in non-stationary environments and highlight when value-based methods may be preferable under milder non-stationarity. Overall, this work advances the understanding of provable, practical policy-optimization approaches for non-stationary RL with function approximation.

Abstract

We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs). In this setting, both the reward function and the transition kernel are linear with respect to the given feature maps and are allowed to vary over time, as long as their respective parameter variations do not exceed certain variation budgets. We propose the \underline{p}eriodically \underline{r}estarted \underline{o}ptimistic \underline{p}olicy \underline{o}ptimization algorithm (PROPO), which is an optimistic policy optimization algorithm with linear function approximation. PROPO features two mechanisms: sliding-window-based policy evaluation and periodic-restart-based policy improvement, which are tailored for policy optimization in a non-stationary environment. In addition, only utilizing the technique of sliding window, we propose a value-iteration algorithm. We establish dynamic upper bounds for the proposed methods and a minimax lower bound which shows the (near-) optimality of the proposed methods. To our best knowledge, PROPO is the first provably efficient policy optimization algorithm that handles non-stationarity.

Paper Structure

This paper contains 43 sections, 22 theorems, 205 equations, 2 figures, 1 table, 7 algorithms.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related Work
  4. Notation
  5. Preliminaries
  6. Non-stationary MDPs
  7. Model Assumptions
  8. Minimax Lower Bound
  9. Algorithm and Theory
  10. PROPO
  11. SW-LSVI-UCB
  12. Regret Analysis
  13. Regret Analysis without Assumption \ref{['assumption:orthonormal']}
  14. Computational Complexity
  15. Parameter-Free Algorithm
  16. ...and 28 more sections

Key Result

Theorem 3.1

Fix $\Delta > 0$, $H > 0$, $d \geq 2$, and $T = \Omega(d^{5/2}\Delta H^{1/2})$. Then, there exists a non-stationary linear kernel MDP with a $d$-dimensional feature map and maximum total variation budget $\Delta$, such that, where $\mathbb{A}$ denotes the learning algorithm which has only access to the bandit feedback.

Figures (2)

  • Figure 1: Comparisons of different methods on cumulative reward under two different environments. The results are averaged over 10 trials and the error bars show the standard deviations. The reward function changes every $100$ episodes in the left subfigure, whereas the reward function changes every $50$ episodes in the right subfigure.
  • Figure 2: The hard-to-learn linear kernel MDP constructed in the proof of Theorem \ref{['thm:lower:bound']}. Note that the probability of state $x_0$ to state $x_1$ depends on the choice of action $a$.

Theorems & Definitions (23)

  • Theorem 3.1: Minimax lower bound
  • Remark 3.2
  • Theorem 4.2: Upper bound for Algorithm \ref{['alg:2']}
  • Theorem 4.3: Informal
  • Theorem 4.4: Upper bound for Algorithm \ref{['alg:4']}
  • Theorem 4.5: Upper bound for Algorithm \ref{['alg:2']}
  • Theorem 4.6: Informal
  • Theorem 4.7: Upper bound for Algorithm \ref{['alg:4']}
  • Theorem 5.1: Upper bound for Algorithm \ref{['alg:B-SW-LSVI-UCB']}
  • Lemma 6.1: One-Step Descent
  • ...and 13 more