Heterogeneity in Multi-Agent Reinforcement Learning

TL;DR

This work defines five agent-level heterogeneity types within the POMG framework and introduces heterogeneity distance, a representation-learning-based metric, to quantify differences between agents. It then presents HetDPS, a dynamic parameter sharing algorithm that clusters agents by heterogeneity and periodically reassigns sharing networks through bipartite matching, improving interpretability and adaptability in MARL. Case studies and experiments on PMS and SMAC show the method identifies environment- and policy-related heterogeneity, with HetDPS achieving competitive performance while using consistent hyperparameters across tasks. Overall, the paper provides a principled toolkit for measuring, interpreting, and leveraging heterogeneity to design more robust and scalable MARL systems.

Abstract

Heterogeneity is a fundamental property in multi-agent reinforcement learning (MARL), which is closely related not only to the functional differences of agents, but also to policy diversity and environmental interactions. However, the MARL field currently lacks a rigorous definition and deeper understanding of heterogeneity. This paper systematically discusses heterogeneity in MARL from the perspectives of definition, quantification, and utilization. First, based on an agent-level modeling of MARL, we categorize heterogeneity into five types and provide mathematical definitions. Second, we define the concept of heterogeneity distance and propose a practical quantification method. Third, we design a heterogeneity-based multi-agent dynamic parameter sharing algorithm as an example of the application of our methodology. Case studies demonstrate that our method can effectively identify and quantify various types of agent heterogeneity. Experimental results show that the proposed algorithm, compared to other parameter sharing baselines, has better interpretability and stronger adaptability. The proposed methodology will help the MARL community gain a more comprehensive and profound understanding of heterogeneity, and further promote the development of practical algorithms.

Figures (9)

• Figure 1: The method of measuring heterogeneity distance based on representation learning.
• Figure 2: The scenario illustration and heterogeneity distance matrices. In v1, the observations of agents from different groups are shuffled in different orders. In v2, the max speeds of agents are different. In v3, one group of agents applies repulsive force to surrounding entities, while the other attractive force. In v4, agents need to move to different landmarks. In v5, both the observations and objectives of agents are heterogeneous. In v6, all the above properties are heterogeneous. Below each scenario illustration, the corresponding heterogeneity distance matrices are shown. Specifically, Obs-Het, Response-Het, Effect-Het, and Objective-Het correspond to observation / response transition / effect transition / objective heterogeneity, respectively.
• Figure 3: Meta-transition heterogeneity and policy heterogeneity distance matrices during training in our case study.
• Figure 4: The method of multi-agent dynamic parameter sharing algorithm based on heterogeneity quantification.
• Figure 5: Results on Partical-based Multi-agent Spreading.