HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning

TL;DR

HCPO tackles scalable coordination in cooperative MARL by introducing a Hierarchical Conductor-based Policy Optimization framework. It augments joint policy expressivity through a central conductor that samples instructions $M$ and a conductor-conditioned joint policy $\boldsymbol{\pi}_{mar}(\boldsymbol{a}|s) = \mathbb{E}_{M}[\boldsymbol{\pi}(\boldsymbol{a}|s,M)]$ with a factorization $\boldsymbol{\pi}(\boldsymbol{a}|s,M) = \prod_i \pi^i( a^i|s,M)$. It then derives a two-level monotonic improvement update, decoupling the conductor’s instruction policy from agents’ policies, with KL-based constraints and a decomposition of $\mathrm{D}_{\mathrm{KL}}$ into $\mathrm{D}_{\mathrm{KL}}^{\max}$ terms, ensuring monotonic $J(\boldsymbol{\pi}_{mar})$ improvements. Centralized training uses a virtual conductor, and after training, a distillation step yields decentralized execution via local conductors. Empirical results on SMAC, MA-MuJoCo, and MPE show HCPO outperforms competitive baselines in cooperative efficiency and stability.

Abstract

In cooperative Multi-Agent Reinforcement Learning (MARL), efficient exploration is crucial for optimizing the performance of joint policy. However, existing methods often update joint policies via independent agent exploration, without coordination among agents, which inherently constrains the expressive capacity and exploration of joint policies. To address this issue, we propose a conductor-based joint policy framework that directly enhances the expressive capacity of joint policies and coordinates exploration. In addition, we develop a Hierarchical Conductor-based Policy Optimization (HCPO) algorithm that instructs policy updates for the conductor and agents in a direction aligned with performance improvement. A rigorous theoretical guarantee further establishes the monotonicity of the joint policy optimization process. By deploying local conductors, HCPO retains centralized training benefits while eliminating inter-agent communication during execution. Finally, we evaluate HCPO on three challenging benchmarks: StarCraftII Multi-agent Challenge, Multi-agent MuJoCo, and Multi-agent Particle Environment. The results indicate that HCPO outperforms competitive MARL baselines regarding cooperative efficiency and stability.

Figures (13)

• Figure 1: Visualization of conductor-based instructional impact in multi-agent learning: Blue and red dots represent players from opposing teams, with the blue team conductor providing strategic instructions.
• Figure 2: The overall framework of HCPO. (a) Centralized training: A two-level policy update mechanism with a virtual centralized conductor is proposed, leveraging well-designed advantage functions. (b) Policy update for agents: Here, local agents' policies denoted by orange ellipses are optimized through sequential updates, and local conductors' policies denoted by blue rectangles are optimized through the cross-entropy method. During this iteration, policies with shaded outlines represent updated versions, while those without shading indicate unmodified ones. (c) Decentralized execution: HCPO enables agents to make decisions based only on local information.
• Figure 3: Performance comparison on SMAC. With the conductor-based joint policy enhancing learning efficiency, HCPO reliably outperforms all baselines.
• Figure 4: Effective coordination in SMAC on the 3s5z map: A visual analysis of agent strategies.
• Figure 5: Comparative evaluation on MA-MuJoCo.

Theorems & Definitions (13)

• Lemma 1
• Lemma 2
• Proposition 1
• Theorem 1
• Lemma \ref{prop1}
• proof
• Lemma \ref{lemma1}
• proof
• Proposition \ref{prop22}
• proof