Provable Cooperative Multi-Agent Exploration for Reward-Free MDPs
Idan Barnea, Orin Levy, Yishay Mansour
TL;DR
The paper addresses reward-free exploration in cooperative multi-agent reinforcement learning for unknown, tabular finite-horizon MDPs, focusing on the trade-off between the number of learning phases ${\rho}$ and the per-phase agent count ${m}$. It introduces MARFE, a phase-based exploration algorithm that uses a reachability threshold ${\beta}$ to concentrate sampling on informative states, and proves that running in exactly ${H}$ phases with ${m = \tilde{O}(S^5 H^6 A (\log(SHA/\delta)+S)/\epsilon^2)}$ agents per phase yields an ${\epsilon}$-accurate dynamics such that ${V^*_{P,r}-V^{\widehat{\pi}_r}_{P,r} \le \epsilon}$ for all reward functions ${r}$ with high probability. A matching lower bound shows that when the number of learning phases is significantly smaller than the horizon, any algorithm requires at least ${m = \Omega(A^{H/\rho}/\rho)}$ agents per phase, highlighting a sharp horizon-driven transition between polynomial and exponential resource regimes. The results advance understanding of how parallel data collection can accelerate reward-free planning in multi-agent systems and inform the design of distributed data-gathering strategies in practice. The work also outlines directions for improving sample-efficiency and extending to larger or continuous state spaces.
Abstract
We study cooperative multi-agent reinforcement learning in the setting of reward-free exploration, where multiple agents jointly explore an unknown MDP in order to learn its dynamics (without observing rewards). We focus on a tabular finite-horizon MDP and adopt a phased learning framework. In each learning phase, multiple agents independently interact with the environment. More specifically, in each learning phase, each agent is assigned a policy, executes it, and observes the resulting trajectory. Our primary goal is to characterize the tradeoff between the number of learning phases and the number of agents, especially when the number of learning phases is small. Our results identify a sharp transition governed by the horizon $H$. When the number of learning phases equals $H$, we present a computationally efficient algorithm that uses only $\tilde{O}(S^6 H^6 A / ε^2)$ agents to obtain an $ε$ approximation of the dynamics (i.e., yields an $ε$-optimal policy for any reward function). We complement our algorithm with a lower bound showing that any algorithm restricted to $ρ< H$ phases requires at least $A^{H/ρ}$ agents to achieve constant accuracy. Thus, we show that it is essential to have an order of $H$ learning phases if we limit the number of agents to be polynomial.