Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation

TL;DR

This work tackles the sample inefficiency of multi-agent reinforcement learning (MARL) by introducing a scenario-independent representation that unifies local and global observations into fixed-size inputs, enabling a single deep policy to operate across different SMAC scenarios. The framework combines Influence Maps (AIM/MAIM) for state representation with a generalized, nearest-enemy-focused action policy, and augments learning with Curriculum Transfer Learning (CTL) to progressively transfer knowledge from simple to more complex tasks. Empirical results in the StarCraft Multi-Agent Challenge show significant improvements in learning speed and final performance for both intra- and inter-agent transfer, with CTL yielding substantial gains on heterogeneous tasks like 2s3z. The approach promises scalable, rapid adaptation of MARL agents in dynamic MAS settings and points to future work in expanding scenario coverage and integrating recurrent architectures.

Abstract

Multi-Agent Reinforcement Learning (MARL) algorithms are widely adopted in tackling complex tasks that require collaboration and competition among agents in dynamic Multi-Agent Systems (MAS). However, learning such tasks from scratch is arduous and may not always be feasible, particularly for MASs with a large number of interactive agents due to the extensive sample complexity. Therefore, reusing knowledge gained from past experiences or other agents could efficiently accelerate the learning process and upscale MARL algorithms. In this study, we introduce a novel framework that enables transfer learning for MARL through unifying various state spaces into fixed-size inputs that allow one unified deep-learning policy viable in different scenarios within a MAS. We evaluated our approach in a range of scenarios within the StarCraft Multi-Agent Challenge (SMAC) environment, and the findings show significant enhancements in multi-agent learning performance using maneuvering skills learned from other scenarios compared to agents learning from scratch. Furthermore, we adopted Curriculum Transfer Learning (CTL), enabling our deep learning policy to progressively acquire knowledge and skills across pre-designed homogeneous learning scenarios organized by difficulty levels. This process promotes inter- and intra-agent knowledge transfer, leading to high multi-agent learning performance in more complicated heterogeneous scenarios.

Figures (7)

• Figure 1: Transfer Between Reinforcement Learning Tasks
• Figure 2: Single Agent's Sight Range in SMAC Scenario
• Figure 3: Transfer Learning Model Representation for Single Unit
• Figure 4: A Sample $19\times19$ Heatmap Generated from Local Observation on $8m$
• Figure 5: Curriculum Transfer Learning Architecture