Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning

Na Li, Zewu Zheng, Wei Ni, Hangguan Shan, Wenjie Zhang, Xinyu Li

TL;DR

This work tackles sample-efficient offline reinforcement learning in robust two-player zero-sum Markov games by introducing RTZ-VI-LCB, a model-based, optimistic value-iteration algorithm tailored for tabular RTZMGs. It integrates a Bernstein-style penalty to handle distribution shifts from partial data and employs a two-stage subsampling scheme to reduce data dependencies, enabling near-optimal sample complexity guarantees. The authors establish an information-theoretic lower bound and a data-driven upper bound, demonstrating optimality in state and action dimensions and analyzing performance under varying uncertainty levels. They also extend the framework to multi-agent robust MGs, offering a scalable approach to offline robust self-play and validating results with numerical experiments. Overall, the paper sets a new benchmark for offline RTZMGs and provides a principled foundation for robust, sample-efficient multi-agent learning from restricted data.

Abstract

Multi-agent reinforcement learning (MARL), as a thriving field, explores how multiple agents independently make decisions in a shared dynamic environment. Due to environmental uncertainties, policies in MARL must remain robust to tackle the sim-to-real gap. We focus on robust two-player zero-sum Markov games (TZMGs) in offline settings, specifically on tabular robust TZMGs (RTZMGs). We propose a model-based algorithm (\textit{RTZ-VI-LCB}) for offline RTZMGs, which is optimistic robust value iteration combined with a data-driven Bernstein-style penalty term for robust value estimation. By accounting for distribution shifts in the historical dataset, the proposed algorithm establishes near-optimal sample complexity guarantees under partial coverage and environmental uncertainty. An information-theoretic lower bound is developed to confirm the tightness of our algorithm's sample complexity, which is optimal regarding both state and action spaces. To the best of our knowledge, RTZ-VI-LCB is the first to attain this optimality, sets a new benchmark for offline RTZMGs, and is validated experimentally.

Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning

TL;DR

This work tackles sample-efficient offline reinforcement learning in robust two-player zero-sum Markov games by introducing RTZ-VI-LCB, a model-based, optimistic value-iteration algorithm tailored for tabular RTZMGs. It integrates a Bernstein-style penalty to handle distribution shifts from partial data and employs a two-stage subsampling scheme to reduce data dependencies, enabling near-optimal sample complexity guarantees. The authors establish an information-theoretic lower bound and a data-driven upper bound, demonstrating optimality in state and action dimensions and analyzing performance under varying uncertainty levels. They also extend the framework to multi-agent robust MGs, offering a scalable approach to offline robust self-play and validating results with numerical experiments. Overall, the paper sets a new benchmark for offline RTZMGs and provides a principled foundation for robust, sample-efficient multi-agent learning from restricted data.

Abstract

Multi-agent reinforcement learning (MARL), as a thriving field, explores how multiple agents independently make decisions in a shared dynamic environment. Due to environmental uncertainties, policies in MARL must remain robust to tackle the sim-to-real gap. We focus on robust two-player zero-sum Markov games (TZMGs) in offline settings, specifically on tabular robust TZMGs (RTZMGs). We propose a model-based algorithm (\textit{RTZ-VI-LCB}) for offline RTZMGs, which is optimistic robust value iteration combined with a data-driven Bernstein-style penalty term for robust value estimation. By accounting for distribution shifts in the historical dataset, the proposed algorithm establishes near-optimal sample complexity guarantees under partial coverage and environmental uncertainty. An information-theoretic lower bound is developed to confirm the tightness of our algorithm's sample complexity, which is optimal regarding both state and action spaces. To the best of our knowledge, RTZ-VI-LCB is the first to attain this optimality, sets a new benchmark for offline RTZMGs, and is validated experimentally.

Paper Structure

This paper contains 75 sections, 15 theorems, 239 equations, 1 figure, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Contribution
  3. Related Work
  4. Finite-sample studies of standard TZMGs.
  5. Robustness in MARL.
  6. Single-agent robust offline RL.
  7. Problem Formulation
  8. Uncertainty set with two-player $(s,a,b)$-rectangularity.
  9. Offline dataset.
  10. Robust value functions.
  11. Robust Bellman equations.
  12. Optimal robust policy.
  13. Robust Nash equilibrium.
  14. Our Goal
  15. Algorithm Design
  16. ...and 60 more sections

Key Result

Lemma 3.1

The dataset produced by the two-stage subsampling method is distributionally identical to $\mathcal{D}_0$ with probability of at least $1-8\delta$, where $\{N_h(s,a,b)\}$ are independent of the sample transitions in $\mathcal{D}^0$ and obey: $\forall (h,s,a,b)\in [H]\times \mathcal{S}\times\mathcal{

Figures (1)

  • Figure 1: The performances of RTZ-VI-LCB and RTZ-VI in the stochastic TZMG problem.

Theorems & Definitions (17)

  • Lemma 3.1
  • Definition 4.1: Robust unilateral clipped concentrability
  • Theorem 4.2: Upper bound for RTZ-VI-LCB
  • Remark 4.3
  • Theorem 4.4: Lower bound for RTZMGs
  • Theorem 4.5: Upper bound for Multi-RTZ-VI-LCB
  • Lemma B.1
  • Lemma D.1
  • Lemma D.2
  • Lemma D.3
  • ...and 7 more