Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games
Tong Yang, Bo Dai, Lin Xiao, Yuejie Chi
TL;DR
This work addresses sample-efficient exploration in multi-agent reinforcement learning within Markov games. It introduces VMG, a model-based framework that uses value-incentivized regularization to bias the model toward parameters with higher collective best-response values, enabling simultaneous uncoupled updates and compatibility with function approximation. The authors prove near-optimal regret bounds for both two-player zero-sum matrix games and finite-horizon multi-player general-sum Markov games under linear function approximation, with extensions to infinite-horizon settings and connections to symmetric matrix games, bandits, and single-agent MDPs. The results suggest VMG is practical for large-scale MARL and could influence real-world multi-agent systems by reducing reliance on explicit uncertainty quantification while retaining strong theoretical guarantees.
Abstract
Multi-agent reinforcement learning (MARL) lies at the heart of a plethora of applications involving the interaction of a group of agents in a shared unknown environment. A prominent framework for studying MARL is Markov games, with the goal of finding various notions of equilibria in a sample-efficient manner, such as the Nash equilibrium (NE) and the coarse correlated equilibrium (CCE). However, existing sample-efficient approaches either require tailored uncertainty estimation under function approximation, or careful coordination of the players. In this paper, we propose a novel model-based algorithm, called VMG, that incentivizes exploration via biasing the empirical estimate of the model parameters towards those with a higher collective best-response values of all the players when fixing the other players' policies, thus encouraging the policy to deviate from its current equilibrium for more exploration. VMG is oblivious to different forms of function approximation, and permits simultaneous and uncoupled policy updates of all players. Theoretically, we also establish that VMG achieves a near-optimal regret for finding both the NEs of two-player zero-sum Markov games and CCEs of multi-player general-sum Markov games under linear function approximation in an online environment, which nearly match their counterparts with sophisticated uncertainty quantification.