Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty

TL;DR

This work tackles robustness to environmental uncertainty in multi‑agent reinforcement learning by formulating finite‑horizon distributionally robust Markov games (RMGs) with agent‑wise, $(s,a)$‑rectangular TV uncertainty. It introduces the distributionally robust Nash value iteration (DR‑NVI), a model‑based algorithm that leverages non‑adaptive sampling from a generative model to compute robust equilibria (NE/CE/CCE) with provable finite‑sample guarantees. The authors establish a matching information‑theoretic lower bound, showing near‑optimal dependence on the state space size $S$, horizon $H$, joint action size $\prod_i A_i$, target accuracy $\varepsilon$, and robustness levels $\{\sigma_i\}$; in the single‑agent reduction, the results are minimax‑optimal for robust MDPs. Overall, the paper provides the first near‑optimal finite‑sample guarantees for learning robust equilibria in multi‑agent settings and demonstrates the efficacy of robust optimization techniques in mitigating sim‑to‑real transfer issues in MARL.

Abstract

To overcome the sim-to-real gap in reinforcement learning (RL), learned policies must maintain robustness against environmental uncertainties. While robust RL has been widely studied in single-agent regimes, in multi-agent environments, the problem remains understudied -- despite the fact that the problems posed by environmental uncertainties are often exacerbated by strategic interactions. This work focuses on learning in distributionally robust Markov games (RMGs), a robust variant of standard Markov games, wherein each agent aims to learn a policy that maximizes its own worst-case performance when the deployed environment deviates within its own prescribed uncertainty set. This results in a set of robust equilibrium strategies for all agents that align with classic notions of game-theoretic equilibria. Assuming a non-adaptive sampling mechanism from a generative model, we propose a sample-efficient model-based algorithm (DRNVI) with finite-sample complexity guarantees for learning robust variants of various notions of game-theoretic equilibria. We also establish an information-theoretic lower bound for solving RMGs, which confirms the near-optimal sample complexity of DRNVI with respect to problem-dependent factors such as the size of the state space, the target accuracy, and the horizon length.

Figures (4)

• Figure 1: A two-player general-sum Markov game modeling preventing illegal fishing. (a) shows the state space (circles) and the simplified transitions; the fisherman arrives at distinct states by executing different Nash equilibrium solutions $\pi_A$ (from city A) or $\pi_B$ (from city B). (b) in two slightly different environments (city A versus city B), it shows the solutions $\pi_A, \pi_B$ of the standard game, and the consistent solution robust Nash$\pi_{rob}$ of a robust variant of the game (detailed in Appendix \ref{['proof:solution-for-example']}).
• Figure 2: Distributionally robust equilibrium value iteration ( DR-NVI).
• Figure 3: Illustration of the sample complexity of DR-NVI with respect to the uncertainty levels $\sigma_1$ and $\sigma_2$ for two-player RMGs, where we only highlight the dependency with respect to the horizon length $H$.
• Figure 4: (a) shows the transition kernels of the game at each time step $h$. (b) illustrates the immediate reward function of two agents.

Theorems & Definitions (11)

• Theorem 1: Upper bound for DR-NVI
• Theorem 2: Lower bound for solving robust MGs
• Lemma 1: Lemma 4, shi2023curious
• Lemma 2
• proof
• Lemma 3
• proof
• Lemma 4
• proof
• Lemma 8