Sample-Efficient Distributionally Robust Multi-Agent Reinforcement Learning via Online Interaction
Zain Ulabedeen Farhat, Debamita Ghosh, George K. Atia, Yue Wang
TL;DR
This paper pioneers the study of online learning in DRMGs, where agents learn directly from environmental interactions without prior data and introduces the Multiplayer Optimistic Robust Nash Value Iteration (MORNAVI) algorithm, the first provable guarantees for this setting.
Abstract
Well-trained multi-agent systems can fail when deployed in real-world environments due to model mismatches between the training and deployment environments, caused by environment uncertainties including noise or adversarial attacks. Distributionally Robust Markov Games (DRMGs) enhance system resilience by optimizing for worst-case performance over a defined set of environmental uncertainties. However, current methods are limited by their dependence on simulators or large offline datasets, which are often unavailable. This paper pioneers the study of online learning in DRMGs, where agents learn directly from environmental interactions without prior data. We introduce the Multiplayer Optimistic Robust Nash Value Iteration (MORNAVI) algorithm and provide the first provable guarantees for this setting. Our theoretical analysis demonstrates that the algorithm achieves low regret and efficiently finds the optimal robust policy for uncertainty sets measured by Total Variation divergence and Kullback-Leibler divergence. These results establish a new, practical path toward developing truly robust multi-agent systems.