Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pausing Policy Learning in Non-stationary Reinforcement Learning

Hyunin Lee, Ming Jin, Javad Lavaei, Somayeh Sojoudi

TL;DR

It is shown that strategically pausing decision updates yields better overall performance by effectively managing aleatoric uncertainty, and that a non-zero policy hold duration provides a sharper upper bound on the dynamic regret.

Abstract

Real-time inference is a challenge of real-world reinforcement learning due to temporal differences in time-varying environments: the system collects data from the past, updates the decision model in the present, and deploys it in the future. We tackle a common belief that continually updating the decision is optimal to minimize the temporal gap. We propose forecasting an online reinforcement learning framework and show that strategically pausing decision updates yields better overall performance by effectively managing aleatoric uncertainty. Theoretically, we compute an optimal ratio between policy update and hold duration, and show that a non-zero policy hold duration provides a sharper upper bound on the dynamic regret. Our experimental evaluations on three different environments also reveal that a non-zero policy hold duration yields higher rewards compared to continuous decision updates.

Pausing Policy Learning in Non-stationary Reinforcement Learning

TL;DR

It is shown that strategically pausing decision updates yields better overall performance by effectively managing aleatoric uncertainty, and that a non-zero policy hold duration provides a sharper upper bound on the dynamic regret.

Abstract

Real-time inference is a challenge of real-world reinforcement learning due to temporal differences in time-varying environments: the system collects data from the past, updates the decision model in the present, and deploys it in the future. We tackle a common belief that continually updating the decision is optimal to minimize the temporal gap. We propose forecasting an online reinforcement learning framework and show that strategically pausing decision updates yields better overall performance by effectively managing aleatoric uncertainty. Theoretically, we compute an optimal ratio between policy update and hold duration, and show that a non-zero policy hold duration provides a sharper upper bound on the dynamic regret. Our experimental evaluations on three different environments also reveal that a non-zero policy hold duration yields higher rewards compared to continuous decision updates.
Paper Structure (23 sections, 16 theorems, 54 equations, 14 figures, 3 algorithms)

This paper contains 23 sections, 16 theorems, 54 equations, 14 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Problem Statement
  4. Method
  5. Theoretical Analysis
  6. Regret analysis
  7. Theoretical insight
  8. Numerical analysis of theoretical insights
  9. Experiments
  10. Future $Q$ value estimator
  11. Goal switching cliffworld
  12. Mujoco environment
  13. Conclusion
  14. Related works
  15. Algorithms
  16. ...and 8 more sections

Key Result

Corollary 3.2

For arbitrary time instances $t_1,t_2 \in [0,T)$ satisfying $t_1<t_2$, there exist constants $\alpha_r,\alpha_p > 1$ such that $\bar{B}_p(t_1,t_2) \leq B^{\text{max}}_p \alpha_{p}^{t_2-t_1}$ and $\bar{B}_r(t_1,t_2) \leq B^{\text{max}}_r \alpha_{r}^{t_2-t_1}$ hold.

Figures (14)

  • Figure 1: (a) Non-stationary bandit setting, (b) conservative policy, (c) pessimistic policy
  • Figure 2: Parallel process of policy learning and data collection.
  • Figure 3: Optimal solutions for $\min_{G_m,N_m} \bar{B}(t_m,t_{m+1})$ are $(G_m^*,N_m^*) = (4,2), (2,4)$. (a) $G_m=4,N_m=2$. (b) $G_m=6,N_m=0$
  • Figure 4: $\mathfrak{R}^{\text{env}}_m$ upper bound with different environmental hyperparameters. $\blacklozenge$ denotes the minimum of each function graph. (a) $\alpha_1 / \alpha_2 \in \{ 0.98, 0.99, 1.0, 1.01, 1.02\}$. (b) $B_1^{\text{max}} / B_2^{\text{max}} \in \{ 0.94, 0.97, 1.0, 1.03, 1.06\}$.
  • Figure 5: $\mathfrak{R}^{\text{env}}_{m}+\mathfrak{R}^{\pi}_{m}$ upper bound with different $C_{\mathfrak{R}}$ ratios and learning rates. $\blacklozenge$ denotes the minimum of each function graph. (a) $C^{\mathfrak{R}}_m \in \{ 0, 0.86, 0.92, 0.95 \}$. (b) Learning rates $\in \{ 0.01, 0.1, 0.3, 0.7, 0.99\}$.
  • ...and 9 more figures

Theorems & Definitions (30)

  • Corollary 3.2: Exponential order cumulative variation budget
  • Definition 3.3: Stationary environment
  • Definition 3.4: Non-stationary environment
  • Proposition 4.1: Linear forecasting method with bounded $l_2$ norm
  • Proposition 4.2: Past uncertainty with sample complexity pmlr-v125-qu20a
  • Corollary 4.3: Maximum forecasting error bound
  • Lemma 5.1: Policy update regret
  • Lemma 5.2: Policy hold regret
  • Theorem 5.3: Dynamic regret
  • Lemma 5.4: Optimal $N^*_m, G^*_m$ for $\mathfrak{R}^\pi_m$
  • ...and 20 more