Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning

Zijun Chen, Shengbo Wang, Nian Si

TL;DR

This work advances distributionally robust reinforcement learning under the average-reward criterion by establishing the first finite-sample guarantees for DR-AMDPs in the tabular setting under KL and f_k divergences. It introduces two model-based algorithms—Reduction to DR-DMDP and Anchored DR-AMDP—that achieve near-optimal sample complexity with a uniform ergodicity assumption on the nominal MDP and without requiring prior knowledge of mixing times. The authors develop stability conditions for the uncertainty sets and show the anchored approach can match the reduction's performance, both yielding a ilde{O}(|S||A| t_{mix}^2 ε^{-2}) sample complexity for robust policy and average reward estimation with small δ. Numerical experiments, including a Hard MDP, corroborate the n^{-1/2} convergence rate and demonstrate practical viability on larger problems.

Abstract

Motivated by practical applications where stable long-term performance is critical-such as robotics, operations research, and healthcare-we study the problem of distributionally robust (DR) average-reward reinforcement learning. We propose two algorithms that achieve near-optimal sample complexity. The first reduces the problem to a DR discounted Markov decision process (MDP), while the second, Anchored DR Average-Reward MDP, introduces an anchoring state to stabilize the controlled transition kernels within the uncertainty set. Assuming the nominal MDP is uniformly ergodic, we prove that both algorithms attain a sample complexity of $\widetilde{O}\left(|\mathbf{S}||\mathbf{A}| t_{\mathrm{mix}}^2\varepsilon^{-2}\right)$ for estimating the optimal policy as well as the robust average reward under KL and $f_k$-divergence-based uncertainty sets, provided the uncertainty radius is sufficiently small. Here, $\varepsilon$ is the target accuracy, $|\mathbf{S}|$ and $|\mathbf{A}|$ denote the sizes of the state and action spaces, and $t_{\mathrm{mix}}$ is the mixing time of the nominal MDP. This represents the first finite-sample convergence guarantee for DR average-reward reinforcement learning. We further validate the convergence rates of our algorithms through numerical experiments.

Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning

TL;DR

This work advances distributionally robust reinforcement learning under the average-reward criterion by establishing the first finite-sample guarantees for DR-AMDPs in the tabular setting under KL and f_k divergences. It introduces two model-based algorithms—Reduction to DR-DMDP and Anchored DR-AMDP—that achieve near-optimal sample complexity with a uniform ergodicity assumption on the nominal MDP and without requiring prior knowledge of mixing times. The authors develop stability conditions for the uncertainty sets and show the anchored approach can match the reduction's performance, both yielding a ilde{O}(|S||A| t_{mix}^2 ε^{-2}) sample complexity for robust policy and average reward estimation with small δ. Numerical experiments, including a Hard MDP, corroborate the n^{-1/2} convergence rate and demonstrate practical viability on larger problems.

Abstract

Motivated by practical applications where stable long-term performance is critical-such as robotics, operations research, and healthcare-we study the problem of distributionally robust (DR) average-reward reinforcement learning. We propose two algorithms that achieve near-optimal sample complexity. The first reduces the problem to a DR discounted Markov decision process (MDP), while the second, Anchored DR Average-Reward MDP, introduces an anchoring state to stabilize the controlled transition kernels within the uncertainty set. Assuming the nominal MDP is uniformly ergodic, we prove that both algorithms attain a sample complexity of for estimating the optimal policy as well as the robust average reward under KL and -divergence-based uncertainty sets, provided the uncertainty radius is sufficiently small. Here, is the target accuracy, and denote the sizes of the state and action spaces, and is the mixing time of the nominal MDP. This represents the first finite-sample convergence guarantee for DR average-reward reinforcement learning. We further validate the convergence rates of our algorithms through numerical experiments.
Paper Structure (25 sections, 42 theorems, 341 equations, 4 figures, 3 tables, 3 algorithms)

This paper contains 25 sections, 42 theorems, 341 equations, 4 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Preliminaries
  4. Markov Decision Processes
  5. Uniform Ergodicity
  6. Distributionally Robust Discounted-Reward and Average-Reward MDPs
  7. DR-AMDP: Algorithms and Sample Complexity Upper Bound
  8. Reduction to DR-DMDP
  9. Anchored DR-AMDP
  10. Numerical Experiments
  11. Conclusion and Future Work
  12. Notations and Basic Properties
  13. Uniform Ergodicity of the KL Uncertainty Set
  14. Properties of the Bellman Operator: KL-Case
  15. Sample Complexity Analysis: KL Uncertainty Set
  16. ...and 10 more sections

Key Result

Proposition 3.6

If $\mathcal{P}$ is uniformly ergodic, then $g_{\mathcal{P}}^*(s)\equiv g_{\mathcal{P}}^*$ is constant in $s\in\mathbf{S}$. Moreover, a solution $(g, v)$ of $v(s) = \mathcal{T}^*(v)(s) - g^*(s)$ for all $s\in\mathbf{S}$ satisfies $g(s) = g_{\mathcal{P}}^*$ for all $s,s'\in\mathbf{S}$. Moreover, the

Figures (4)

  • Figure 1: Transition diagram of the hard MDP instance in wang_optimal_2023.
  • Figure 2: Comparative numerical experiments on (a-b) Algorithm \ref{['alg:reduction_to_dmdp']} and (c-d) Algorithm \ref{['alg:anchored_amdp']} for the hard MDP instance, demonstrating $\varepsilon$-dependence under different divergence measures.
  • Figure 3: KL-divergence case for Algorithm \ref{['alg:reduction_to_dmdp']} and \ref{['alg:anchored_amdp']}
  • Figure :

Theorems & Definitions (82)

  • Definition 3.1
  • Definition 3.2
  • Definition 3.3: Uniformly Ergodic MDP
  • Definition 3.4
  • Definition 3.5
  • Proposition 3.6: Theorem 7 of wang_robust_2023
  • Definition 4.1
  • Proposition 4.2
  • Theorem 4.3
  • Theorem 4.4
  • ...and 72 more