Refined Sample Complexity for Markov Games with Independent Linear Function Approximation
Yan Dai, Qiwen Cui, Simon S. Du
TL;DR
This work addresses the sample-efficiency challenge in multi-agent Markov Games with independent linear function approximation by refining the AVLPR framework with data-dependent pessimistic gap estimators. It introduces action-dependent bonuses and Magnitude-Reduced Estimators, combined with Adaptive Freedman inequalities and covariance-matrix concentration, to achieve the optimal $O(T^{-1/2})$ convergence while avoiding polynomial dependence on the action size $A_{ ext{max}}$. The resulting algorithm attains a high-probability $ ext{CCE}$ guarantee with a final sample complexity of $ ilde{O}(m^4 d^5 H^6 ilde{oldsymbol{ ext{epsilon}}}^{-2})$, marking the first approach to circumvent the curse of multi-agents under independent linear function approximation. The framework blends ideas from adversarial contextual bandits, linear MDP theory, and stochastic matrix concentration to deliver practical, scalable MARL guarantees. This has potential impact for large-scale, multi-agent decision-making systems where joint state-action spaces are intractable and function approximation is essential.
Abstract
Markov Games (MG) is an important model for Multi-Agent Reinforcement Learning (MARL). It was long believed that the "curse of multi-agents" (i.e., the algorithmic performance drops exponentially with the number of agents) is unavoidable until several recent works (Daskalakis et al., 2023; Cui et al., 2023; Wang et al., 2023). While these works resolved the curse of multi-agents, when the state spaces are prohibitively large and (linear) function approximations are deployed, they either had a slower convergence rate of $O(T^{-1/4})$ or brought a polynomial dependency on the number of actions $A_{\max}$ -- which is avoidable in single-agent cases even when the loss functions can arbitrarily vary with time. This paper first refines the AVLPR framework by Wang et al. (2023), with an insight of designing *data-dependent* (i.e., stochastic) pessimistic estimation of the sub-optimality gap, allowing a broader choice of plug-in algorithms. When specialized to MGs with independent linear function approximations, we propose novel *action-dependent bonuses* to cover occasionally extreme estimation errors. With the help of state-of-the-art techniques from the single-agent RL literature, we give the first algorithm that tackles the curse of multi-agents, attains the optimal $O(T^{-1/2})$ convergence rate, and avoids $\text{poly}(A_{\max})$ dependency simultaneously.