Adaptive parameter sharing for multi-agent reinforcement learning

TL;DR

AdaPS introduces a brain-inspired adaptive parameter sharing scheme for multi-agent reinforcement learning. It uses a VAE to extract agent identity, clusters agents by identity, and generates cluster-specific subnetworks via a fixed mapping mask, enabling diversity without increasing training parameters. The approach achieves competitive or superior performance across multiple environments while maintaining similar parameter counts to standard sharing methods, demonstrating improved scalability and sample efficiency. This framework offers a practical route to handle heterogeneity in large-scale MARL without the parameter blow-up typically associated with selective sharing.

Abstract

Parameter sharing, as an important technique in multi-agent systems, can effectively solve the scalability issue in large-scale agent problems. However, the effectiveness of parameter sharing largely depends on the environment setting. When agents have different identities or tasks, naive parameter sharing makes it difficult to generate sufficiently differentiated strategies for agents. Inspired by research pertaining to the brain in biology, we propose a novel parameter sharing method. It maps each type of agent to different regions within a shared network based on their identity, resulting in distinct subnetworks. Therefore, our method can increase the diversity of strategies among different agents without introducing additional training parameters. Through experiments conducted in multiple environments, our method has shown better performance than other parameter sharing methods.

Figures (4)

• Figure 1: Tasks mapping to specific region in human brains.
• Figure 2: The AdaPS framework.
• Figure 3: The mean test return on three environments. The numbers in bracket represent the number of agents in each type.
• Figure 4: Relative model size of different algorithms.