Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nearly Minimax Optimal Regret for Learning Linear Mixture Stochastic Shortest Path

Qiwei Di, Jiafan He, Dongruo Zhou, Quanquan Gu

TL;DR

This work tackles learning in stochastic shortest path problems with linear mixture transitions by introducing LEVIS$^{++}$, a computationally efficient algorithm that uses extended value iteration (DEVI) together with a variance-aware, high-order moment–based confidence framework (HOME). The method eliminates the need for a strictly positive cost or known horizon, achieving a regret of $\tilde{O}(d B_* \sqrt{K})$, which nearly matches the known lower bound $\Omega(d B_* \sqrt{K})$ and thus is nearly minimax optimal. The key innovations are variance-aware weights that combine estimated variance with uncertainty terms, and a multi-level moment estimator that propagates high-order information through weighted regressions. Together, these techniques enable horizon-free regret bounds in linear mixture SSPs, with practical implications for scalable RL in large state-action spaces.

Abstract

We study the Stochastic Shortest Path (SSP) problem with a linear mixture transition kernel, where an agent repeatedly interacts with a stochastic environment and seeks to reach certain goal state while minimizing the cumulative cost. Existing works often assume a strictly positive lower bound of the cost function or an upper bound of the expected length for the optimal policy. In this paper, we propose a new algorithm to eliminate these restrictive assumptions. Our algorithm is based on extended value iteration with a fine-grained variance-aware confidence set, where the variance is estimated recursively from high-order moments. Our algorithm achieves an $\tilde{\mathcal O}(dB_*\sqrt{K})$ regret bound, where $d$ is the dimension of the feature mapping in the linear transition kernel, $B_*$ is the upper bound of the total cumulative cost for the optimal policy, and $K$ is the number of episodes. Our regret upper bound matches the $Ω(dB_*\sqrt{K})$ lower bound of linear mixture SSPs in Min et al. (2022), which suggests that our algorithm is nearly minimax optimal.

Nearly Minimax Optimal Regret for Learning Linear Mixture Stochastic Shortest Path

TL;DR

This work tackles learning in stochastic shortest path problems with linear mixture transitions by introducing LEVIS, a computationally efficient algorithm that uses extended value iteration (DEVI) together with a variance-aware, high-order moment–based confidence framework (HOME). The method eliminates the need for a strictly positive cost or known horizon, achieving a regret of , which nearly matches the known lower bound and thus is nearly minimax optimal. The key innovations are variance-aware weights that combine estimated variance with uncertainty terms, and a multi-level moment estimator that propagates high-order information through weighted regressions. Together, these techniques enable horizon-free regret bounds in linear mixture SSPs, with practical implications for scalable RL in large state-action spaces.

Abstract

We study the Stochastic Shortest Path (SSP) problem with a linear mixture transition kernel, where an agent repeatedly interacts with a stochastic environment and seeks to reach certain goal state while minimizing the cumulative cost. Existing works often assume a strictly positive lower bound of the cost function or an upper bound of the expected length for the optimal policy. In this paper, we propose a new algorithm to eliminate these restrictive assumptions. Our algorithm is based on extended value iteration with a fine-grained variance-aware confidence set, where the variance is estimated recursively from high-order moments. Our algorithm achieves an regret bound, where is the dimension of the feature mapping in the linear transition kernel, is the upper bound of the total cumulative cost for the optimal policy, and is the number of episodes. Our regret upper bound matches the lower bound of linear mixture SSPs in Min et al. (2022), which suggests that our algorithm is nearly minimax optimal.
Paper Structure (22 sections, 25 theorems, 105 equations, 1 figure, 2 tables, 3 algorithms)

This paper contains 22 sections, 25 theorems, 105 equations, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. The Proposed Algorithm
  5. Algorithm Description
  6. Comparison with $\text{LEVIS}^+$ min2022learning
  7. Our Key Techniques
  8. Main Results
  9. Proof Sketch
  10. Analysis of DEVI
  11. Regret Decomposition
  12. Conclusions
  13. Notations
  14. Numerical Simulations
  15. Analysis of Algorithm
  16. ...and 7 more sections

Key Result

Lemma 3.3

Suppose that Assumption assumption:policy holds and for every improper policy $\pi$, there exists at least one state $s$, such that $V^{\pi}(s) = \infty$, then there exists an optimal policy $\pi^*$, which is a stationary, deterministic, and proper. What's more, $V^* = V^{\pi^*}$ is the unique solut

Figures (1)

  • Figure 1: The plot shows the average regret (i.e. $R_K/K$) and compares the implementation results of Algorithm \ref{['algorithm1']} and LEVIS in min2022learning on the SSP instance described in Appendix B with $\lambda = 1, \rho = 0$ and failing probability 0.01.

Theorems & Definitions (26)

  • Lemma 3.3: bertsekas1991analysisyu2013boundednesstarbouriech2021stochastic
  • Theorem 4.1: Theorem G.2 in min2022learning
  • Theorem 5.1: Known $c_{\text{min}}$
  • Remark 5.2
  • Theorem 5.3: Known $T_*$
  • Lemma 6.1
  • Lemma 6.2
  • Lemma 6.3
  • Lemma 6.4
  • Lemma 3.1
  • ...and 16 more