Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IPPO Learns the Game, Not the Team: A Study on Generalization in Heterogeneous Agent Teams

Ryan LeRoy, Jack Kolb

TL;DR

The paper investigates whether self-play trained MARL policies generalize to coordination with unseen, heterogeneous teammates. It introduces Rotating Policy Training (RPT), which alternates among PPO, A2C, and DQN policies during training in the HeMAC environment, and compares it to a standard Independent PPO baseline. Results show that IPPO generalizes to novel teammates as well as RPT, while RPT offers no clear performance edge and suffers from lower sample efficiency due to rotation. The findings suggest that, at least in HeMAC, PPO learns underlying game dynamics and robust coordination rather than memorizing teammate-specific conventions, and that RPT primarily serves as a diagnostic tool for understanding robustness and learning dynamics.

Abstract

Multi-Agent Reinforcement Learning (MARL) is commonly deployed in settings where agents are trained via self-play with homogeneous teammates, often using parameter sharing and a single policy architecture. This opens the question: to what extent do self-play PPO agents learn general coordination strategies grounded in the underlying game, compared to overfitting to their training partners' behaviors? This paper investigates the question using the Heterogeneous Multi-Agent Challenge (HeMAC) environment, which features distinct Observer and Drone agents with complementary capabilities. We introduce Rotating Policy Training (RPT), an approach that rotates heterogeneous teammate policies of different learning algorithms during training, to expose the agent to a broader range of partner strategies. When playing alongside a withheld teammate policy (DDQN), we find that RPT achieves similar performance to a standard self-play baseline, IPPO, where all agents were trained sharing a single PPO policy. This result indicates that in this heterogeneous multi-agent setting, the IPPO baseline generalizes to novel teammate algorithms despite not experiencing teammate diversity during training. This shows that a simple IPPO baseline may possess the level of generalization to novel teammates that a diverse training regimen was designed to achieve.

IPPO Learns the Game, Not the Team: A Study on Generalization in Heterogeneous Agent Teams

TL;DR

The paper investigates whether self-play trained MARL policies generalize to coordination with unseen, heterogeneous teammates. It introduces Rotating Policy Training (RPT), which alternates among PPO, A2C, and DQN policies during training in the HeMAC environment, and compares it to a standard Independent PPO baseline. Results show that IPPO generalizes to novel teammates as well as RPT, while RPT offers no clear performance edge and suffers from lower sample efficiency due to rotation. The findings suggest that, at least in HeMAC, PPO learns underlying game dynamics and robust coordination rather than memorizing teammate-specific conventions, and that RPT primarily serves as a diagnostic tool for understanding robustness and learning dynamics.

Abstract

Multi-Agent Reinforcement Learning (MARL) is commonly deployed in settings where agents are trained via self-play with homogeneous teammates, often using parameter sharing and a single policy architecture. This opens the question: to what extent do self-play PPO agents learn general coordination strategies grounded in the underlying game, compared to overfitting to their training partners' behaviors? This paper investigates the question using the Heterogeneous Multi-Agent Challenge (HeMAC) environment, which features distinct Observer and Drone agents with complementary capabilities. We introduce Rotating Policy Training (RPT), an approach that rotates heterogeneous teammate policies of different learning algorithms during training, to expose the agent to a broader range of partner strategies. When playing alongside a withheld teammate policy (DDQN), we find that RPT achieves similar performance to a standard self-play baseline, IPPO, where all agents were trained sharing a single PPO policy. This result indicates that in this heterogeneous multi-agent setting, the IPPO baseline generalizes to novel teammate algorithms despite not experiencing teammate diversity during training. This shows that a simple IPPO baseline may possess the level of generalization to novel teammates that a diverse training regimen was designed to achieve.

Paper Structure

This paper contains 21 sections, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cooperative MARL and CTDE
  4. Ad-hoc Teamwork and Teammate Modeling
  5. Population and League-Based Training
  6. Method
  7. Policy Architectures
  8. Training Procedure
  9. Experiments
  10. Environment: HeMAC
  11. Baselines
  12. Training Protocol
  13. Generalization Evaluation
  14. Results
  15. Discussion
  16. ...and 6 more sections

Figures (3)

  • Figure 1: The HeMAC environment with one Observer (plane) and one Drone. Observers have a wide vision cone, while Drones have limited vision but can physically intercept the target.
  • Figure 2: The plot showing training reward curves for the RPT and PPO policies. These rewards are then plotted based on number of agent timesteps trained. To do this the RPT rewards were down-sampled by a factor of 3 to account for the random switching.
  • Figure 3: The plot showing reward when evaluated with mixed in novel teammate policies. These rewards are then plotted based on number of agent timesteps trained. To do this the RPT rewards were down-sampled by a factor of 3 to account for the random switching.