Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning

TL;DR

This work addresses the challenge of efficient exploration in cooperative multi-agent reinforcement learning by introducing CoopTS-PHE and CoopTS-LMC, two Thompson Sampling-inspired strategies embedded in a unified framework for randomized exploration in parallel MDPs. The authors provide rigorous regret and communication guarantees in linear, homogeneous settings and extend the theory to misspecified and heterogeneous scenarios, while demonstrating practical gains on N-chain, Super Mario Bros, and Building Energy Systems tasks. The combination of perturbed-history noise and Langevin Monte Carlo sampling yields a computation-friendly alternative to full posterior methods, with demonstrated robustness and a natural pathway to federated RL. Overall, the paper delivers the first theoretical and empirical validation of randomized exploration in cooperative MARL, offering a scalable, communication-aware approach for large-scale, distributed RL systems.

Abstract

We present the first study on provably efficient randomized exploration in cooperative multi-agent reinforcement learning (MARL). We propose a unified algorithm framework for randomized exploration in parallel Markov Decision Processes (MDPs), and two Thompson Sampling (TS)-type algorithms, CoopTS-PHE and CoopTS-LMC, incorporating the perturbed-history exploration (PHE) strategy and the Langevin Monte Carlo exploration (LMC) strategy, respectively, which are flexible in design and easy to implement in practice. For a special class of parallel MDPs where the transition is (approximately) linear, we theoretically prove that both CoopTS-PHE and CoopTS-LMC achieve a $\widetilde{\mathcal{O}}(d^{3/2}H^2\sqrt{MK})$ regret bound with communication complexity $\widetilde{\mathcal{O}}(dHM^2)$, where $d$ is the feature dimension, $H$ is the horizon length, $M$ is the number of agents, and $K$ is the number of episodes. This is the first theoretical result for randomized exploration in cooperative MARL. We evaluate our proposed method on multiple parallel RL environments, including a deep exploration problem (i.e., $N$-chain), a video game, and a real-world problem in energy systems. Our experimental results support that our framework can achieve better performance, even under conditions of misspecified transition models. Additionally, we establish a connection between our unified framework and the practical application of federated learning.

Figures (14)

• Figure 1: Comparison among different exploration strategies in different environments. (a)-(b): $N$-chain with $N=25$. (c)-(d): Super Mario Bros. All results are averaged over 10 runs and the shaded area represents the standard deviation.
• Figure 2: Evaluation performance at Tampa (hot humid) in building energy systems. All results are averaged over 10 runs.
• Figure 3: The N-Chain environment bootstrappeddqn.
• Figure 4: Comparison among different exploration strategies in $N$-chain with $N=25$. All results are averaged over 10 runs.
• Figure 5: Rewards with averaged over 10 independent runs for different numbers of agents among algorithms without communication. Note that when $m = 1$, one agent indicates a centralized setting.

Theorems & Definitions (53)

• Definition 4.1: Linear MDP jin2020provably
• Theorem 4.2
• Remark 4.3
• Theorem 4.4
• Remark 4.5
• Lemma 4.6
• Remark 4.7
• Definition 4.8: Misspecified Parallel MDPs
• Remark 4.9
• Theorem 4.10: Misspecified Regret Bound for CoopTS-PHE