Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving Tractable Minimax Optimal Regret in Average Reward MDPs

Victor Boone, Zihan Zhang

TL;DR

This work addresses learning in average-reward MDPs by introducing PMEVI-DT, a tractable OFU-based method that attains minimax optimal regret $\tilde{O}(\sqrt{sp(h^*)\,S A\,T})$ without requiring knowledge of the bias span $sp(h^*)$. The core idea is the PMEVI subroutine, which performs bias-constrained planning via a projection onto a bias-confidence region and a beta-mitigated extended Bellman operator to tightly control bias dynamics. The framework supports incorporating prior bias information to tighten the bias region and can be embedded into existing optimistic algorithms to improve regret bounds; a Weissman-type confidence-region implementation yields a polynomial per-step complexity. Theoretical guarantees are complemented by river-swim experiments showing favorable regret and burn-in behavior, highlighting practical impact for average-reward RL in large or unknown environments. Overall, the paper closes the gap between minimax optimal regret and tractable computation in average-reward MDPs and provides a versatile tool for bias-aware planning.

Abstract

In recent years, significant attention has been directed towards learning average-reward Markov Decision Processes (MDPs). However, existing algorithms either suffer from sub-optimal regret guarantees or computational inefficiencies. In this paper, we present the first tractable algorithm with minimax optimal regret of $\widetilde{\mathrm{O}}(\sqrt{\mathrm{sp}(h^*) S A T})$, where $\mathrm{sp}(h^*)$ is the span of the optimal bias function $h^*$, $S \times A$ is the size of the state-action space and $T$ the number of learning steps. Remarkably, our algorithm does not require prior information on $\mathrm{sp}(h^*)$. Our algorithm relies on a novel subroutine, Projected Mitigated Extended Value Iteration (PMEVI), to compute bias-constrained optimal policies efficiently. This subroutine can be applied to various previous algorithms to improve regret bounds.

Achieving Tractable Minimax Optimal Regret in Average Reward MDPs

TL;DR

This work addresses learning in average-reward MDPs by introducing PMEVI-DT, a tractable OFU-based method that attains minimax optimal regret without requiring knowledge of the bias span . The core idea is the PMEVI subroutine, which performs bias-constrained planning via a projection onto a bias-confidence region and a beta-mitigated extended Bellman operator to tightly control bias dynamics. The framework supports incorporating prior bias information to tighten the bias region and can be embedded into existing optimistic algorithms to improve regret bounds; a Weissman-type confidence-region implementation yields a polynomial per-step complexity. Theoretical guarantees are complemented by river-swim experiments showing favorable regret and burn-in behavior, highlighting practical impact for average-reward RL in large or unknown environments. Overall, the paper closes the gap between minimax optimal regret and tractable computation in average-reward MDPs and provides a versatile tool for bias-aware planning.

Abstract

In recent years, significant attention has been directed towards learning average-reward Markov Decision Processes (MDPs). However, existing algorithms either suffer from sub-optimal regret guarantees or computational inefficiencies. In this paper, we present the first tractable algorithm with minimax optimal regret of , where is the span of the optimal bias function , is the size of the state-action space and the number of learning steps. Remarkably, our algorithm does not require prior information on . Our algorithm relies on a novel subroutine, Projected Mitigated Extended Value Iteration (PMEVI), to compute bias-constrained optimal policies efficiently. This subroutine can be applied to various previous algorithms to improve regret bounds.
Paper Structure (46 sections, 40 theorems, 124 equations, 3 figures, 1 table)

This paper contains 46 sections, 40 theorems, 124 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related works.
  4. Preliminaries
  5. Infinite-horizon MDP.
  6. Weakly-communicating MDPs.
  7. Reinforcement learning.
  8. Algorithm PMEVI-DT
  9. Extended Bellman operators and EVI.
  10. Towards Projected Mitigated EVI.
  11. Projected mitigated extended value iteration (PMEVI)
  12. Building the bias confidence region and its projection operator
  13. Mitigation using finer bias dynamical error
  14. Regret guarantees
  15. Experimental illustrations
  16. ...and 31 more sections

Key Result

Theorem 1

For any $c>0$, provided that the confidence region used by PMEVI-DT satisfy mild regularity conditions (see Assumption assumption:model-confidence-region-assumption:evi-convergence), if $T \ge c^5$, then for every weakly communicating model with bias span less than $c$ and with bias vector within $ in expectation and with high probability. Moreover, if PMEVI-DT runs with the same confidence regi

Figures (3)

  • Figure 1: An overview of PMEVI-DT and its regret analysis. In the above, $\mathfrak{g}_k$ and $\mathfrak{h}_k$ are the optimistic gain and bias functions produced by PMEVI (see Algorithm 2) at episode $k$, and $\hat{p}_{t_k}$ and $\tilde{p}_{t_k}$ are respectively the empirical and optimistic kernel models at episode $k$.
  • Figure 2: (To the left) Running a few algorithms of the literature on $5$-state river-swim and comparing their average regret against their PMEVI variants, obtained by changing calls to the EVI sub-routine to calls to PMEVI. (To the right) Running UCRL2 and PMEVI-DT with the same confidence region that UCRL2 on a $3$-state river-swim. PMEVI-DT is run with prior knowledge $h^*(s_1) \le h^*(s_2) - c \le h^*(s_3) - 2c$ for $c \in \left\{0, 0.5, 1, 1.5, 2\right\}$.
  • Figure 3: The kernel of a $n$-state river-swim.

Theorems & Definitions (60)

  • Theorem 1: Informal
  • Proposition 2
  • Definition 1: Bias difference estimator
  • Lemma 3
  • Lemma 4
  • Theorem 5: Main result
  • Lemma 6: Reward optimism
  • Lemma 7: Navigation error
  • Lemma 8: Empirical bias error
  • Lemma 9: Optimism overshoot
  • ...and 50 more