Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation

TL;DR

This work addresses the challenge of delayed observations in multi-agent reinforcement learning by formalizing the decentralized stochastic individual delay POMDP (DSID-POMDP) and introducing the Rainbow Delay Compensation (RDC) framework. RDC combines a compensator (Echo/Flash) to reconstruct delay-free observations, a delay-reconciled critic for centralized training, curriculum learning for actors, and knowledge distillation from a low-delay teacher to guide policy learning under delays. Experiments on MPE and SMAC show that RDC substantially mitigates the negative impact of both fixed and unfixed delays, with several settings approaching ideal delay-free performance and good generalization across delay distributions. The framework is modular and broadly compatible with standard MARL baselines (e.g., VDN/QMIX), and the authors provide source code to facilitate adoption in real-world MAS applications where delayed sensing is inevitable.

Abstract

In real-world multi-agent systems (MASs), observation delays are ubiquitous, preventing agents from making decisions based on the environment's true state. An individual agent's local observation typically comprises multiple components from other agents or dynamic entities within the environment. These discrete observation components with varying delay characteristics pose significant challenges for multi-agent reinforcement learning (MARL). In this paper, we first formulate the decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP) by extending the standard Dec-POMDP. We then propose the Rainbow Delay Compensation (RDC), a MARL training framework for addressing stochastic individual delays, along with recommended implementations for its constituent modules. We implement the DSID-POMDP's observation generation pattern using standard MARL benchmarks, including MPE and SMAC. Experiments demonstrate that baseline MARL methods suffer severe performance degradation under fixed and unfixed delays. The RDC-enhanced approach mitigates this issue, remarkably achieving ideal delay-free performance in certain delay scenarios while maintaining generalizability. Our work provides a novel perspective on multi-agent delayed observation problems and offers an effective solution framework. The source code is available at https://github.com/linkjoker1006/RDC-pymarl.

Figures (21)

• Figure 1: A simple example of extended state and delayed observation in DSID-POMDP. The left side describes the components of $agent_1$'s observation and annotates their delay value distributions. The matrix in the upper right corner shows the contents of $s_{(-1)}$ in the extended state.
• Figure 2: The internal structure of the DA-MARL framework.
• Figure 3: A simplified example illustrating MASs' delay occurrence and compensation process.
• Figure 4: Inputs of compensators with different modes and networks. The figure only illustrates the input formation of a single $agent_i$ at timestep $t$, and for clarity, we omit the subscript $i$. The subscript denotes the timestep in the environment iteration, while the superscript indicates both the $k$-th output of Echo and the $(k+1)$-th input. Inputs of Flash exclude the yellow-highlighted components.
• Figure 5: Workflow of Flash and Echo.