Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Sample-Efficient Robust Reinforcement Learning with Average Reward

Zachary Roch, Chi Zhang, George Atia, Yue Wang

TL;DR

This work addresses robust reinforcement learning under the average-reward criterion with uncertain transitions. It introduces Robust Halpern Iteration (RHI), a model-free algorithm that operates under the weakest standard assumptions (communicating RAMDPs) and requires no prior knowledge of the MDP, achieving a near-optimal sample complexity of $\tilde{O}(SA\mathcal{H}^2/\epsilon^2)$. The method leverages a quotient-space formulation and Halpern iteration to handle a non-contractive robust Bellman operator, along with a novel R-SAMPLE estimator for two uncertainty sets. Empirical results on Garnet demonstrate that RHI reliably converges to the robust average-reward optimum across contamination and $\ell_p$ uncertainty models, validating the theoretical guarantees and practical viability of robust average-reward RL in data-limited settings.

Abstract

Robust reinforcement learning (RL) under the average-reward criterion is essential for long-term decision-making, particularly when the environment may differ from its specification. However, a significant gap exists in understanding the finite-sample complexity of these methods, as most existing work provides only asymptotic guarantees. This limitation hinders their principled understanding and practical deployment, especially in data-limited scenarios. We close this gap by proposing \textbf{Robust Halpern Iteration (RHI)}, a new algorithm designed for robust Markov Decision Processes (MDPs) with transition uncertainty characterized by $\ell_p$-norm and contamination models. Our approach offers three key advantages over previous methods: (1). Weaker Structural Assumptions: RHI only requires the underlying robust MDP to be communicating, a less restrictive condition than the commonly assumed ergodicity or irreducibility; (2). No Prior Knowledge: Our algorithm operates without requiring any prior knowledge of the robust MDP; (3). State-of-the-Art Sample Complexity: To learn an $ε$-optimal robust policy, RHI achieves a sample complexity of $\tilde{\mathcal O}\left(\frac{SA\mathcal H^{2}}{ε^{2}}\right)$, where $S$ and $A$ denote the numbers of states and actions, and $\mathcal H$ is the robust optimal bias span. This result represents the tightest known bound. Our work hence provides essential theoretical understanding of sample efficiency of robust average reward RL.

Provably Sample-Efficient Robust Reinforcement Learning with Average Reward

TL;DR

This work addresses robust reinforcement learning under the average-reward criterion with uncertain transitions. It introduces Robust Halpern Iteration (RHI), a model-free algorithm that operates under the weakest standard assumptions (communicating RAMDPs) and requires no prior knowledge of the MDP, achieving a near-optimal sample complexity of . The method leverages a quotient-space formulation and Halpern iteration to handle a non-contractive robust Bellman operator, along with a novel R-SAMPLE estimator for two uncertainty sets. Empirical results on Garnet demonstrate that RHI reliably converges to the robust average-reward optimum across contamination and uncertainty models, validating the theoretical guarantees and practical viability of robust average-reward RL in data-limited settings.

Abstract

Robust reinforcement learning (RL) under the average-reward criterion is essential for long-term decision-making, particularly when the environment may differ from its specification. However, a significant gap exists in understanding the finite-sample complexity of these methods, as most existing work provides only asymptotic guarantees. This limitation hinders their principled understanding and practical deployment, especially in data-limited scenarios. We close this gap by proposing \textbf{Robust Halpern Iteration (RHI)}, a new algorithm designed for robust Markov Decision Processes (MDPs) with transition uncertainty characterized by -norm and contamination models. Our approach offers three key advantages over previous methods: (1). Weaker Structural Assumptions: RHI only requires the underlying robust MDP to be communicating, a less restrictive condition than the commonly assumed ergodicity or irreducibility; (2). No Prior Knowledge: Our algorithm operates without requiring any prior knowledge of the robust MDP; (3). State-of-the-Art Sample Complexity: To learn an -optimal robust policy, RHI achieves a sample complexity of , where and denote the numbers of states and actions, and is the robust optimal bias span. This result represents the tightest known bound. Our work hence provides essential theoretical understanding of sample efficiency of robust average reward RL.
Paper Structure (16 sections, 18 theorems, 105 equations, 1 figure, 1 table, 3 algorithms)

This paper contains 16 sections, 18 theorems, 105 equations, 1 figure, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and problem formulation
  3. Communicating RAMDPs
  4. Robust Halpern iteration (RHI) for robust AMDPs
  5. Related Work
  6. Comparisons with Prior Results
  7. Other Related work
  8. Experiment results
  9. Conclusion
  10. Preliminaries and Proof Organization
  11. Proof of \ref{['thm:bellman']}
  12. Sampling algorithm
  13. Proofs for RHI
  14. Analysis of RHI
  15. PF-RHI: a parameter-free variant of RHI
  16. ...and 1 more sections

Key Result

Theorem 3.2

Consider a robust AMDP satisfying ass. Then it holds that: (1). The optimal robust average reward $g^*_\mathcal{P}$ is a constant, i.e., $g^*_\mathcal{P}(s_1)=g^*_\mathcal{P}(s_2), \forall s_1\neq s_2$; (2). The robust Bellman equation in eq:bellman q has a solution $(Q^*, g^{*})$, and the solution

Figures (1)

  • Figure 1: Performance of RHI.

Theorems & Definitions (34)

  • Theorem 3.2
  • Lemma 4.1
  • Theorem 4.2
  • Theorem 4.3: Performance of RHI
  • Remark 4.4
  • Theorem B.1
  • proof
  • Lemma D.1: Restatement of \ref{['prop:1']}
  • proof
  • Remark D.2
  • ...and 24 more