Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior

TL;DR

This paper addresses scalable, decentralized MARL under partial observability by proposing Decentralized Partially Observable Mean Field Control (Dec-POMFC). By taking the infinite-agent mean-field limit, it derives a tractable Dec-MFC MDP that links decentralized policies to a centralized training objective, with a dynamic programming principle guaranteeing optimality under Lipschitz policies. Kernel-based mean-field representations enable efficient handling of high-dimensional state-action spaces, and a policy-gradient method (Dec-POMFPPO) provides practical learning with CTDE. Empirically, the approach demonstrates strong performance on collective behavior tasks (e.g., Aggregation, Vicsek, Kuramoto) and matches state-of-the-art MARL benchmarks, while offering theoretical guarantees and scalability to very large agent counts. Overall, the framework advances RL-driven engineering of artificial swarms by marrying mean-field theory with decentralized, partially observable learning.

Abstract

Recent reinforcement learning (RL) methods have achieved success in various domains. However, multi-agent RL (MARL) remains a challenge in terms of decentralization, partial observability and scalability to many agents. Meanwhile, collective behavior requires resolution of the aforementioned challenges, and remains of importance to many state-of-the-art applications such as active matter physics, self-organizing systems, opinion dynamics, and biological or robotic swarms. Here, MARL via mean field control (MFC) offers a potential solution to scalability, but fails to consider decentralized and partially observable systems. In this paper, we enable decentralized behavior of agents under partial information by proposing novel models for decentralized partially observable MFC (Dec-POMFC), a broad class of problems with permutation-invariant agents allowing for reduction to tractable single-agent Markov decision processes (MDP) with single-agent RL solution. We provide rigorous theoretical results, including a dynamic programming principle, together with optimality guarantees for Dec-POMFC solutions applied to finite swarms of interest. Algorithmically, we propose Dec-POMFC-based policy gradient methods for MARL via centralized training and decentralized execution, together with policy gradient approximation guarantees. In addition, we improve upon state-of-the-art histogram-based MFC by kernel methods, which is of separate interest also for fully observable MFC. We evaluate numerically on representative collective behavior tasks such as adapted Kuramoto and Vicsek swarming models, being on par with state-of-the-art MARL. Overall, our framework takes a step towards RL-based engineering of artificial collective behavior via MFC.

Figures (24)

• Figure 1: A: Partially-observable Vicsek problem: agents must align headings (arrows), but observe only partial information (e.g. heading distribution in grey circle for orange agent). B: The decentralized model as a graphical model (grey: observed variables). C: In centralized training, we also observe the mean field, guiding the learning of upper-level actions $\check \pi$. D: The solved limiting MDP.
• Figure 2: Three steps of approximation (mean field limit, open-loop control, and MDP reformulation) allow us to reformulate the broad class of MFC-type Dec-POMDP to a tractable Dec-MFC MDP.
• Figure 3: Dec-POMFPPO training curves (episode return) with shaded standard deviation over $3$ seeds for $N=200$ in (a) Aggregation; Vicsek on a (b): torus; (c): Möbius strip; (d): projective plane; (e): Klein bottle; and (f) Kuramoto on a torus.
• Figure 4: Training curves (episode return) with shaded standard deviation over $3$ seeds and $N=200$, in (a) Aggregation (box), (b) Vicsek (torus), (c) Kuramoto (torus). For comparison, we also plot the best return averaged over $3$ seeds for Dec-POMFPPO in Figure \ref{['fig:training']} (MF).
• Figure 5: The performance of the best of $3$ Dec-POMFPPO policies transferred to $N$-agent systems (in blue, error bars for $95\%$ confidence interval), averaged over $50$ episodes, and compared against the performance in the training system (in red). Problems (a)-(f) and training are as in Figure \ref{['fig:training']}.

Theorems & Definitions (35)

• Theorem 1
• Corollary 1
• Proposition 1
• Corollary 2
• Theorem 2
• Proposition 2
• Corollary 3
• Proposition 3
• Theorem 3
• proof : Proof of Theorem \ref{['thm:mf_conv']}