QTypeMix: Enhancing Multi-Agent Cooperative Strategies through Heterogeneous and Homogeneous Value Decomposition

TL;DR

QTypeMix tackles heterogeneous cooperative MARL by introducing a type-aware, dual-layer value function factorization that splits the joint value into homogeneous per-type components and a subsequent heterogeneous aggregation. A Type Embedding Extractor derives type-sensitive features from agents' history and is trained with a TE loss to encourage clear type separation, while attention and hypernetworks implement the two mixing stages. Empirical results on SMAC and SMACv2 demonstrate state-of-the-art performance, especially in settings with many agent types, and show faster convergence than baselines. The approach provides a practical, type-driven framework that reduces reliance on external domain knowledge for role assignment and can be extended to policy-based MARL in future work.

Abstract

In multi-agent cooperative tasks, the presence of heterogeneous agents is familiar. Compared to cooperation among homogeneous agents, collaboration requires considering the best-suited sub-tasks for each agent. However, the operation of multi-agent systems often involves a large amount of complex interaction information, making it more challenging to learn heterogeneous strategies. Related multi-agent reinforcement learning methods sometimes use grouping mechanisms to form smaller cooperative groups or leverage prior domain knowledge to learn strategies for different roles. In contrast, agents should learn deeper role features without relying on additional information. Therefore, we propose QTypeMix, which divides the value decomposition process into homogeneous and heterogeneous stages. QTypeMix learns to extract type features from local historical observations through the TE loss. In addition, we introduce advanced network structures containing attention mechanisms and hypernets to enhance the representation capability and achieve the value decomposition process. The results of testing the proposed method on 14 maps from SMAC and SMACv2 show that QTypeMix achieves state-of-the-art performance in tasks of varying difficulty.

Figures (9)

• Figure 1: Diagram of our straightforward type-wise value decomposition method (QTypeMix-B).
• Figure 2: Diagram of hyper policy network.
• Figure 3: Diagram of additional type information extraction.
• Figure 4: Diagram of QTypeMix.
• Figure 5: Test battle win rate curves of the six algorithms during the training process on 4 SMAC maps.