RMIO: A Model-Based MARL Framework for Scenarios with Observation Loss in Some Agents

TL;DR

RMIO tackles the challenge of observation loss in multi-agent reinforcement learning by jointly reconstructing missing observations with a world model and refining them via a correction block that aggregates inter-agent information. It preserves CTDE in normal operation and uses minimal communication during observation loss, reducing coordination overhead. The approach combines reward smoothing, a dual-layer replay buffer, and an RNN-augmented policy to improve asymptotic convergence and policy robustness, with strong empirical gains on SMAC and MaMuJoCo benchmarks. These results demonstrate a practical, scalable path to robust, sample-efficient MARL under adverse observation conditions, with potential impact on real-world multi-agent systems facing intermittent sensing and communication failures.

Abstract

In recent years, model-based reinforcement learning (MBRL) has emerged as a solution to address sample complexity in multi-agent reinforcement learning (MARL) by modeling agent-environment dynamics to improve sample efficiency. However, most MBRL methods assume complete and continuous observations from each agent during the inference stage, which can be overly idealistic in practical applications. A novel model-based MARL approach called RMIO is introduced to address this limitation, specifically designed for scenarios where observation is lost in some agent. RMIO leverages the world model to reconstruct missing observations, and further reduces reconstruction errors through inter-agent information integration to ensure stable multi-agent decision-making. Secondly, unlike CTCE methods such as MAMBA, RMIO adopts the CTDE paradigm in standard environment, and enabling limited communication only when agents lack observation data, thereby reducing reliance on communication. Additionally, RMIO improves asymptotic performance through strategies such as reward smoothing, a dual-layer experience replay buffer, and an RNN-augmented policy model, surpassing previous work. Our experiments conducted in both the SMAC and MaMuJoCo environments demonstrate that RMIO outperforms current state-of-the-art approaches in terms of asymptotic convergence performance and policy robustness, both in standard mission settings and in scenarios involving observation loss.

Figures (8)

• Figure 1: At each step a random subset of agents has a probability of experiencing observation loss. The process of completing and correcting observations by RMIO (only $m$ agents get observations).
• Figure 2: Network composition and inference process of correction block. The scenario in the figure assumes that there are $m$ agents obtaining accurate observations, and $n-m$ agents cannot obtain any observation values.
• Figure 3: Double Experience Replay Buffer structure.
• Figure 4: The whole reasoning process of RMIO facing observation loss(only $m$ agents get accurate $\{o_t^i\}_{i=1}^m$) at time step $t$. RMIO first communicate to synchronize the historical $\{\boldsymbol{o}_{t-i},\boldsymbol{a}_{t-i}\}_{i=t}^1$ among agents, getting historical status information $\boldsymbol{h}_t$. Based on $\boldsymbol{h}_t$, the prior model and observation predictor can reconstruct the missing $\{o_t^i\}_{i=n-m+1}^n$, getting $\hat{o}_t^i$. At last, the correction block use the partial accurate $\{{o}^i_t\}_{i=1}^m$ to correct the estimated observation$\{\hat{o}_t^i\}_{i=m+1}^n\}$, getting $\boldsymbol{\hat{o}}_t=\{\{{o}^i_t\}_{i=1}^m,\{\hat{\hat{o}}^i_t\}_{i=m+1}^n\}$. Thus, the agents can take action based on the estimated $\boldsymbol{\hat{o}}_t$.
• Figure 5: Taking agent 3 as an example, the light colored parts in the figure represent the status information of other agents that agent 3 cannot access. At time $T-2$, each agent executes in a distributed manner, and agent 3 can only access local state information. At time $T-1$, agent 3 experiences observation loss. At this time, the missing observation information is supplemented and corrected through communication with the world model to support agent 3's decision-making. Through communication, agent 3 can access some state information of other agents. At time $T$, agent 3 experiences observation loss again, and it can be considered that the two observation losses are in a "close" proximity state. Agent 3 does not need to communicate at this time, but directly uses the state information of other agents obtained after $T-1$ communication to make prior estimates of the observed state at time $T$ and the actions of other agents, thereby completing and correcting the missing observed state at time $T$. This process does not require further communication and is completed locally by agent 3.

Theorems & Definitions (1)

• Definition 1