Multi-agent Markov Entanglement

TL;DR

This work establishes a theory for value decomposition in multi-agent reinforcement learning by introducing Markov entanglement, a measure of inter-agent dependence in the joint transition dynamics. It proves that exact Q-value decomposition is possible if and only if the joint transition is separable, and shows that the decomposition error in general MDPs is bounded by the entanglement measure under several distances (TV, ATV, and μ-weighted variants). The authors demonstrate sublinear decomposition error for index policies in restless multi-armed bandits via mean-field analysis and provide practical, data-driven methods to estimate Markov entanglement as a proxy for decomposition feasibility. They also extend the theory to multi-agent settings, discuss extensions to exogenous information and factored/weakly coupled MDPs, and validate the framework with simulations in RMAB and ride-hailing contexts. Overall, the work offers a rigorous foundation and practical toolkit for when and how value decomposition can be trusted in large-scale MARL systems.

Abstract

Value decomposition has long been a fundamental technique in multi-agent dynamic programming and reinforcement learning (RL). Specifically, the value function of a global state $(s_1,s_2,\ldots,s_N)$ is often approximated as the sum of local functions: $V(s_1,s_2,\ldots,s_N)\approx\sum_{i=1}^N V_i(s_i)$. This approach traces back to the index policy in restless multi-armed bandit problems and has found various applications in modern RL systems. However, the theoretical justification for why this decomposition works so effectively remains underexplored. In this paper, we uncover the underlying mathematical structure that enables value decomposition. We demonstrate that a multi-agent Markov decision process (MDP) permits value decomposition if and only if its transition matrix is not "entangled" -- a concept analogous to quantum entanglement in quantum physics. Drawing inspiration from how physicists measure quantum entanglement, we introduce how to measure the "Markov entanglement" for multi-agent MDPs and show that this measure can be used to bound the decomposition error in general multi-agent MDPs. Using the concept of Markov entanglement, we proved that a widely-used class of index policies is weakly entangled and enjoys a sublinear $\mathcal O(\sqrt{N})$ scale of decomposition error for $N$-agent systems. Finally, we show how Markov entanglement can be efficiently estimated in practice, providing practitioners with an empirical proxy for the quality of value decomposition.

Figures (4)

• Figure 1: Circulant RMAB under an index policy. Left: empirical estimation of Markov entanglement $E_{1}({\bm{P}}^\pi_{1:N})$. Right:$\mu$-weighted decomposition error.
• Figure 2: A ride-hailing simulator. Left: empirical estimation of Markov entanglement $E_{1}({\bm{P}}^\pi_{1:N})$. Right:$\mu$-weighted decomposition error divided by the global value $\|Q^\pi_{1:N}\|_\mu$.
• Figure 3: Value Decomposition error in circulant RMAB under an index policy. Left: Relative error, $\left\lVert\textrm{decomposition error}\right\rVert_\mu/\left\lVert Q^\pi_{1:N}\right\rVert_\mu$. Right: Different errors in RMAB with $900$ agents: empirical estimation of Markov entanglement (blue); $\mu^\pi_{1:N}$-weighted decomposition error (green); the true measure of Markov estimated with $T=10N$ samples (red dashed line).
• Figure 4: Taxi zone map of Manhattan NYCTLCData

Theorems & Definitions (48)

• Definition 1: Two-agent Markov Entanglement
• Theorem 1
• Example 1: Shared Randomness
• Definition 2: Two-party Quantum Entanglement
• Theorem 2
• Definition 3: Measure of Two-agent Markov Entanglement
• Definition 4: Total Variation Distance between Transition Matrices
• Theorem 3
• Definition 5: Agent-wise Total Variation Distance
• Theorem 4