Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

Lang Feng, Jiahao Lin, Dong Xing, Li Zhang, De Ma, Gang Pan

TL;DR

This work tackles population-population generalization in multi-agent reinforcement learning by introducing Bidirectional Distillation (BiDist), a mixed-play framework that uses a fictitious background population and alternating forward and reverse distillations. Forward distillation emulates past policies to realize implicit self-play, while reverse distillation drives diversity beyond the historical policy space to address outside-space generalization, all without storing policy pools. Theoretical results bound the generalization error via a delta-cover framework and show that BiDist reduces the covering radius, while empirical results on Melting Pot tasks demonstrate improved generalization across cooperative, competitive, and social-dilemma settings and greater policy-space diversification. The approach is resource-efficient and robust, offering a practical path to more generalizable MARL agents with straightforward integration into existing architectures like MAPPO.

Abstract

Population-population generalization is a challenging problem in multi-agent reinforcement learning (MARL), particularly when agents encounter unseen co-players. However, existing self-play-based methods are constrained by the limitation of inside-space generalization. In this study, we propose Bidirectional Distillation (BiDist), a novel mixed-play framework, to overcome this limitation in MARL. BiDist leverages knowledge distillation in two alternating directions: forward distillation, which emulates the historical policies' space and creates an implicit self-play, and reverse distillation, which systematically drives agents towards novel distributions outside the known policy space in a non-self-play manner. In addition, BiDist operates as a concise and efficient solution without the need for the complex and costly storage of past policies. We provide both theoretical analysis and empirical evidence to support BiDist's effectiveness. Our results highlight its remarkable generalization ability across a variety of cooperative, competitive, and social dilemma tasks, and reveal that BiDist significantly diversifies the policy distribution space. We also present comprehensive ablation studies to reinforce BiDist's effectiveness and key success factors. Source codes are available in the supplementary material.

Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

TL;DR

This work tackles population-population generalization in multi-agent reinforcement learning by introducing Bidirectional Distillation (BiDist), a mixed-play framework that uses a fictitious background population and alternating forward and reverse distillations. Forward distillation emulates past policies to realize implicit self-play, while reverse distillation drives diversity beyond the historical policy space to address outside-space generalization, all without storing policy pools. Theoretical results bound the generalization error via a delta-cover framework and show that BiDist reduces the covering radius, while empirical results on Melting Pot tasks demonstrate improved generalization across cooperative, competitive, and social-dilemma settings and greater policy-space diversification. The approach is resource-efficient and robust, offering a practical path to more generalizable MARL agents with straightforward integration into existing architectures like MAPPO.

Abstract

Population-population generalization is a challenging problem in multi-agent reinforcement learning (MARL), particularly when agents encounter unseen co-players. However, existing self-play-based methods are constrained by the limitation of inside-space generalization. In this study, we propose Bidirectional Distillation (BiDist), a novel mixed-play framework, to overcome this limitation in MARL. BiDist leverages knowledge distillation in two alternating directions: forward distillation, which emulates the historical policies' space and creates an implicit self-play, and reverse distillation, which systematically drives agents towards novel distributions outside the known policy space in a non-self-play manner. In addition, BiDist operates as a concise and efficient solution without the need for the complex and costly storage of past policies. We provide both theoretical analysis and empirical evidence to support BiDist's effectiveness. Our results highlight its remarkable generalization ability across a variety of cooperative, competitive, and social dilemma tasks, and reveal that BiDist significantly diversifies the policy distribution space. We also present comprehensive ablation studies to reinforce BiDist's effectiveness and key success factors. Source codes are available in the supplementary material.
Paper Structure (44 sections, 3 theorems, 8 equations, 10 figures, 3 tables, 1 algorithm)

This paper contains 44 sections, 3 theorems, 8 equations, 10 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Setting and Background
  4. Problem Setting.
  5. Training protocol.
  6. Testing protocol.
  7. Bidirectional Distillation
  8. Random Fictitious Population
  9. Forward Distillation
  10. Reverse Distillation
  11. Theoretical Analysis
  12. Policy Iteration
  13. Experiment
  14. Experimental Setup
  15. Generalization Task.
  16. ...and 29 more sections

Key Result

proposition 1

Given the observation $o_i$ and a distilled policy $\pi_{\phi_i}$ that attains the upper bound $\mathcal{H}^{\max}(\pi_{\phi_i}(\cdot|o_i))$ of entropy maximization, there exists an update margin for KL divergence maximization that enables $\pi_{\phi_i}(a^{\text{prefer}}_{\theta_i}|o_i)$ to continue

Figures (10)

  • Figure 1: The training and testing phases of zero-shot co-player generalization task in MARL. FP and BP denote the focal population (blue) and background population (yellow) respectively.
  • Figure 2: Inside-space and outside-space generalization. denotes the zero-shot policy, denotes the current training policy, and denotes the historical training policy.
  • Figure 3: Illustration of BiDist in an 8-agent task. Based on vector $\bm{v}$, agents are divided into two populations: trained (blue) and fictitious (orange) populations. They respectively adopt the learning policies (blue) and the distilled policies (orange) to gather trajectories in the substrate. The distilled policies are updated by forward and reverse distillations alternately at an interval $k_d$. The diagram shows the iterations from $k-k_d$ to $k+k_d$, a complete alternating cycle, where forward distillation occurs at iteration $k$; reverse distillation occurs at iteration $k+k_d$; the remaining iterations do not involve any distillations.
  • Figure 4: Illustrative instances of the testing scenarios (0-2) for the substrate Prisoners Dilemma in the matrix: Arena. The background population is visually emphasized through bounding boxes. Pre-trained background population varies in different scenarios, including the number of agents and decision-making preferences.
  • Figure 5: The t-SNE results of different algorithms on Pure Coordination, Chicken, and Coop Mining tasks. Each sample represents the joint probability distribution of all agents' actions.
  • ...and 5 more figures

Theorems & Definitions (4)

  • proposition 1
  • definition 1
  • lemma 1
  • theorem 1