Interactive Distillation for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This work tackles efficiency and robustness in cooperative multi-agent reinforcement learning by addressing KD bottlenecks when a centralized teacher guides decentralized students. It introduces HINT, a Hierarchical INteractive Teacher-based transfer framework, featuring a centralized HiT-MAC teacher, a heterogeneous HetNet student, and three interacting components: knowledge distillation, pseudo off-policy RL, and DAgger with performance-based filtering. The approach enables the teacher to continuously refine its guidance using student trajectories and to provide outcome-relevant demonstrations, improving adaptation to OOD states and observation mismatches. Empirical results on MARINE and FireCommander show HINT consistently outperforms competitive CTDE and KD baselines by substantial margins (60%–165% in success rate), validating its robustness and practical potential for scalable cooperative MARL.

Abstract

Knowledge distillation (KD) has the potential to accelerate MARL by employing a centralized teacher for decentralized students but faces key bottlenecks. Specifically, there are (1) challenges in synthesizing high-performing teaching policies in complex domains, (2) difficulties when teachers must reason in out-of-distribution (OOD) states, and (3) mismatches between the decentralized students' and the centralized teacher's observation spaces. To address these limitations, we propose HINT (Hierarchical INteractive Teacher-based transfer), a novel KD framework for MARL in a centralized training, decentralized execution setup. By leveraging hierarchical RL, HINT provides a scalable, high-performing teacher. Our key innovation, pseudo off-policy RL, enables the teacher policy to be updated using both teacher and student experience, thereby improving OOD adaptation. HINT also applies performance-based filtering to retain only outcome-relevant guidance, reducing observation mismatches. We evaluate HINT on challenging cooperative domains (e.g., FireCommander for resource allocation, MARINE for tactical combat). Across these benchmarks, HINT outperforms baselines, achieving improvements of 60% to 165% in success rate.

Figures (15)

• Figure 1: Bridging the distribution gap between teacher and student. As task complexity increases, teachers trained offline provide unreliable guidance on student trajectories that diverge from the training distribution; we close this gap via adaptive refinement. (Example environment: MARINE)
• Figure 2: Overview of HINT. (a) A centralized teacher guides decentralized students via three mechanisms: knowledge distillation for student updates, pseudo off-policy RL for teacher refinement, and dataset aggregation with performance-based filtering to support student queries and build a high-quality dataset. (b) The teacher operates hierarchically, with a high-level coordinator assigning subgoals to low-level executors based on a task-specific hierarchical structure. This structure enables temporal abstraction by decoupling strategic and tactical decisions. (c) Each student includes a preprocessing unit, an LSTM encoder for temporal abstraction, and a decoder for action selection, while HetGAT layers enable inter-agent communication.
• Figure 3: Comparison of teacher (blue) and student (red) state distributions projected into a shared latent space for MARINE and FC. Each setting corresponds to student rollouts with a 10–30% success rate. As task complexity increases, the gap between student and teacher distributions widens (measured by KL-divergence), indicating that teachers are increasingly exposed to out-of-distribution (OOD) states.
• Figure 4: Teacher policy ($\pi_T$) is refined using its on-policy rollouts (shaded blue) and off-policy rollouts (shaded orange) from the student ($\pi_S$), with policy gradients corrected via importance sampling.
• Figure 5: Performance-based filter is applied during dataset aggregation. High-quality teacher demonstrations are accepted (green, ✓), while suboptimal ones are rejected (red, ✗).