RL in Markov Games with Independent Function Approximation: Improved Sample Complexity Bound under the Local Access Model

TL;DR

The paper tackles the challenge of efficiently learning equilibria in general-sum Markov games with large state/action spaces by adopting independent linear function approximation per agent to avoid the exponential curse of multi-agency. It introduces Lin-Confident-FTRL, a multi-phase algorithm that uses per-agent core sets and a decentralized Follow-The-Regularized-Leader update to learn $ε$-CCE under a local access model, achieving a near-optimal sample complexity that reduces dependence on the joint action space. The analysis leverages a virtual policy iteration framework to derive bounds under both local and random access, yielding - local-access: $\tilde{O}(\min\{\log(S)/d, \max_i A_i\} d^3 H^6 m^2 \varepsilon^{-2})$ samples (small regime) and $\tilde{O}(m^2 d^5 H^{14} \varepsilon^{-6})$ (large regime), and - random-access: $\tilde{O}(\min\{\varepsilon^{-2} d H^2, \log(S)/d, \max_i A_i\} d^2 H^6 m^2 \varepsilon^{-2})$ samples, with $(\varepsilon+3\nu\sqrt{d}H)$-CCE guarantees in both settings. This work advances scalable MARL with function approximation by decoupling agent-wise estimation, providing practical pathways for decentralized deployment and tighter theoretical guarantees that outperform prior online/local policies on sample complexity.

Abstract

Efficiently learning equilibria with large state and action spaces in general-sum Markov games while overcoming the curse of multi-agency is a challenging problem. Recent works have attempted to solve this problem by employing independent linear function classes to approximate the marginal $Q$-value for each agent. However, existing sample complexity bounds under such a framework have a suboptimal dependency on the desired accuracy $\varepsilon$ or the action space. In this work, we introduce a new algorithm, Lin-Confident-FTRL, for learning coarse correlated equilibria (CCE) with local access to the simulator, i.e., one can interact with the underlying environment on the visited states. Up to a logarithmic dependence on the size of the state space, Lin-Confident-FTRL learns $ε$-CCE with a provable optimal accuracy bound $O(ε^{-2})$ and gets rids of the linear dependency on the action space, while scaling polynomially with relevant problem parameters (such as the number of agents and time horizon). Moreover, our analysis of Linear-Confident-FTRL generalizes the virtual policy iteration technique in the single-agent local planning literature, which yields a new computationally efficient algorithm with a tighter sample complexity bound when assuming random access to the simulator.

Figures (7)

• Figure : Explore$(s,h)$
• Figure : Lin-Confident-FTRL
• Figure : Multi-Agent-Learning
• Figure : Local Sampling$(h, i, s, a, \pi_{-i})$
• Figure : Policy Rollout

Theorems & Definitions (20)

• Lemma 1: yin2022efficient
• Theorem 1
• Theorem 2
• Lemma 2
• Lemma 3
• proof
• Lemma 4
• Lemma 5
• Lemma 6
• proof : Proof of Lemma \ref{['lem-G2-bound']}