Distributionally Robust Online Markov Game with Linear Function Approximation

TL;DR

This work tackles the sim-to-real gap in multi-agent reinforcement learning by formulating a distributionally robust online learning framework for general-sum Markov games with linear function approximation. It establishes a hardness result under a $d$-rectangular uncertainty model and overcomes it via a vanishing minimal value assumption, enabling tractable analysis. The authors introduce DR-CCE-LSI, a least-squares value-iteration style algorithm with per-agent exploration bonuses to compute an $\epsilon$-approximate robust CCE, achieving a regret bound of $O\{dH\min\{H,1/\min_i\sigma_i\}\sqrt{K}\}$ and proving minimax optimality in the feature dimension $d$. Simulations validate robustness against environment shifts and demonstrate improved performance over non-robust baselines, highlighting the practical potential of robust online MARL in uncertain domains.

Abstract

The sim-to-real gap, where agents trained in a simulator face significant performance degradation during testing, is a fundamental challenge in reinforcement learning. Extansive works adopt the framework of distributionally robust RL, to learn a policy that acts robustly under worst case environment shift. Within this framework, our objective is to devise algorithms that are sample efficient with interactive data collection and large state spaces. By assuming d-rectangularity of environment dynamic shift, we identify a fundamental hardness result for learning in online Markov game, and address it by adopting minimum value assumption. Then, a novel least square value iteration type algorithm, DR-CCE-LSI, with exploration bonus devised specifically for multiple agents, is proposed to find an \episilon-approximate robust Coarse Correlated Equilibrium(CCE). To obtain sample efficient learning, we find that: when the feature mapping function satisfies certain properties, our algorithm, DR-CCE-LSI, is able to achieve ε-approximate CCE with a regret bound of O{dHmin{H,1/min{σ_i}}\sqrt{K}}, where K is the number of interacting episodes, H is the horizon length, d is the feature dimension, and \simga_i represents the uncertainty level of player i. Our work introduces the first sample-efficient algorithm for this setting, matches the best result so far in single agent setting, and achieves minimax optimalsample complexity in terms of the feature dimension d. Meanwhile, we also conduct simulation study to validate the efficacy of our algorithm in learning a robust equilibrium.

Figures (4)

• Figure 1: Constructed simulated Markov game
• Figure 2: Comparison of the average reward under different uncertainty level.
• Figure 3: Performance comparison across various players and perturbed sensitivity level.
• Figure 4: The Flow of robust Markov Game in Example \ref{['example:robust markov game lower bound']}. For player i, $s_{good}^i$ represents the favorable state in which the player receives a reward of 1, otherwise 0. The solid line indicates transitions of the nominal transition kernel $P^\theta$ for $\mathcal{MG}_{rob}^\theta$, while dashed line is involved when we consider robustness. For green line, it indicates difference in the transition probability of the Markov game instance.

Theorems & Definitions (38)

• Definition 3.2
• Definition 3.3
• Theorem 4.1
• Proposition 4.3: Equivalence of Optimization under Minimal Value Assumption
• Lemma 4.4
• Theorem 5.1: Instance Dependent Upper Bound
• Theorem 5.2
• Corollary 5.3
• Example B.1
• Lemma B.2