Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sequence Modeling for N-Agent Ad Hoc Teamwork

Caroline Wang, Di Yang Shi, Elad Liebman, Ishan Durugkar, Arrasy Rahman, Peter Stone

TL;DR

NAHT task requires dynamic collaboration with unseen teammates without pre-coordination. The authors propose MAT-NAHT, a centralized transformer-based method that encodes histories of controlled agents to adapt to a variety of unknown teammates in partially observable environments. Empirical results on StarCraft II SMAC show MAT-NAHT achieves better generalization and sample efficiency than the prior POAM baseline, without explicit agent-modeling objectives. This work demonstrates the potential of history-conditioned, centralized transformers for scalable ad hoc teamwork with varying team sizes and compositions.

Abstract

N-agent ad hoc teamwork (NAHT) is a newly introduced challenge in multi-agent reinforcement learning, where controlled subteams of varying sizes must dynamically collaborate with varying numbers and types of unknown teammates without pre-coordination. The existing learning algorithm (POAM) considers only independent learning for its flexibility in dealing with a changing number of agents. However, independent learning fails to fully capture the inter-agent dynamics essential for effective collaboration. Based on our observation that transformers deal effectively with sequences with varying lengths and have been shown to be highly effective for a variety of machine learning problems, this work introduces a centralized, transformer-based method for N-agent ad hoc teamwork. Our proposed approach incorporates historical observations and actions of all controlled agents, enabling optimal responses to diverse and unseen teammates in partially observable environments. Empirical evaluation on a StarCraft II task demonstrates that MAT-NAHT outperforms POAM, achieving superior sample efficiency and generalization, without auxiliary agent-modeling objectives.

Sequence Modeling for N-Agent Ad Hoc Teamwork

TL;DR

NAHT task requires dynamic collaboration with unseen teammates without pre-coordination. The authors propose MAT-NAHT, a centralized transformer-based method that encodes histories of controlled agents to adapt to a variety of unknown teammates in partially observable environments. Empirical results on StarCraft II SMAC show MAT-NAHT achieves better generalization and sample efficiency than the prior POAM baseline, without explicit agent-modeling objectives. This work demonstrates the potential of history-conditioned, centralized transformers for scalable ad hoc teamwork with varying team sizes and compositions.

Abstract

N-agent ad hoc teamwork (NAHT) is a newly introduced challenge in multi-agent reinforcement learning, where controlled subteams of varying sizes must dynamically collaborate with varying numbers and types of unknown teammates without pre-coordination. The existing learning algorithm (POAM) considers only independent learning for its flexibility in dealing with a changing number of agents. However, independent learning fails to fully capture the inter-agent dynamics essential for effective collaboration. Based on our observation that transformers deal effectively with sequences with varying lengths and have been shown to be highly effective for a variety of machine learning problems, this work introduces a centralized, transformer-based method for N-agent ad hoc teamwork. Our proposed approach incorporates historical observations and actions of all controlled agents, enabling optimal responses to diverse and unseen teammates in partially observable environments. Empirical evaluation on a StarCraft II task demonstrates that MAT-NAHT outperforms POAM, achieving superior sample efficiency and generalization, without auxiliary agent-modeling objectives.

Paper Structure

This paper contains 13 sections, 3 figures.

Table of Contents

  1. Introduction & Background
  2. Preliminaries
  3. Training and Evaluation in Practice
  4. Multi-Agent Transformers for N-Agent Ad Hoc Teamwork
  5. Encoder.
  6. Decoder.
  7. Experimental Evaluation
  8. Experimental Setting
  9. Environment Details
  10. Uncontrolled Teammates
  11. Baseline
  12. Results
  13. Discussion

Figures (3)

  • Figure 1: Architecture of the multi-agent transformer for NAHT. Only observations and actions corresponding to the controlled agents (agents $1, \cdots, N$) are provided to the encoder. Similarly, the autoregressive action decoder decodes actions for the controlled agents only.
  • Figure 2: Mean test return and 95% CI on 3sv5z for MAT-NAHT versus Cent Obs POAM when paired with $X_{test}$ teammates, computed over five trials. MAT-NAHT has improved generalization compared to Cent Obs POAM, across all five test teammate types.
  • Figure 3: Sample efficiency curves with a 95% CI on 3sv5z task across five trials. This reflects the performance over $X_{train}$ teammates averaged over the uncontrolled teammate types.