Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards

TL;DR

GPAT addresses zero-shot coordination in ad hoc teams by formulating an ad hoc MMDP and leveraging a library of pretrained policies through generalized policy improvement (GPI) while using learner-specific difference rewards to mitigate distribution shift from unseen teammates. It trains a set of learner policies on source ad hoc teams, learns $Δr$-based value functions, and at test time applies a GPI policy to select actions from the library without online learning, with optional rapid fine-tuning if allowed. Empirical results across foraging, predator-prey, Overcooked, and a real-world TurtleBot3 demonstration show GPAT outperforming robust and type-based baselines and approaching oracle performance when relevant skills exist in the library, while ablations highlight the beneficial role of difference rewards for policy switching. This approach reduces reliance on large pretraining pools and online inference, enabling scalable, data-efficient zero-shot transfer in cooperative multi-agent systems.

Abstract

Real-world multi-agent systems may require ad hoc teaming, where an agent must coordinate with other previously unseen teammates to solve a task in a zero-shot manner. Prior work often either selects a pretrained policy based on an inferred model of the new teammates or pretrains a single policy that is robust to potential teammates. Instead, we propose to leverage all pretrained policies in a zero-shot transfer setting. We formalize this problem as an ad hoc multi-agent Markov decision process and present a solution that uses two key ideas, generalized policy improvement and difference rewards, for efficient and effective knowledge transfer between different teams. We empirically demonstrate that our algorithm, Generalized Policy improvement for Ad hoc Teaming (GPAT), successfully enables zero-shot transfer to new teams in three simulated environments: cooperative foraging, predator-prey, and Overcooked. We also demonstrate our algorithm in a real-world multi-robot setting.

Figures (8)

• Figure 1: Common methods for AHT either select a pretrained policy based on the inferred behavior of the teammate (Type-based) or train a single policy that is robust to potential teammates by generating a diverse training pool (Robust). We instead dynamically select which policy to use at every timestep without inference for ZSC.
• Figure 2: Difference rewards allow the learner to estimate its own contribution to the team reward. We illustrate this in a foraging example where collecting any object gives a team reward of 1. (left) the teammate action results in the reward and (right) the learner action results in the reward. While the team reward is the same in both scenarios, $\Delta r$ reflects the learner's contribution. We use this idea to reduce the effect of the distribution shift induced by new teammates.
• Figure 3: Our method, GPAT, addresses ZSC in AHT through 3 steps: (1) pretraining learner policies, (2) learning difference reward value functions to address the distribution shift induced by new teammates, and (3) dynamically leveraging this library at every time step by applying a GPI policy.
• Figure 4: Multi-agent environments used in our experiments, where the learner agent is green. (a) Circles represent different types of objects agents aim to collect. (b) Agents aim to capture easy (yellow) and hard (red) prey. (c) Agents must coordinate to cook and serve a tomato onion soup. (d) Our multi-robot demonstration uses TurtleBot3 Burgers.
• Figure 5: The average number of times the learner performs various subtasks over an episode, measured using 1000 episodes with 10 replicates. The Robust learner struggles to pot onions and tomatoes, despite learning how to optimally perform both during pretraining.

Theorems & Definitions (1)

• Definition 1: Ad Hoc MMDP