Representation Learning For Efficient Deep Multi-Agent Reinforcement Learning

TL;DR

The paper addresses the persistent challenge of sample inefficiency in deep multi-agent reinforcement learning by introducing MAPO-LSO, a framework that augments MARL with latent-space optimization. It decomposes MA-LSO into MA-TDR, which reconstructs multi-agent transition dynamics using recurrent modeling and predictive representations, and MA-SPL, which enforces self-predictive consistency across agents through masked reconstruction, forward and inverse dynamics, and dedicated MLP heads. The approach can be integrated with existing MARL algorithms (e.g., MAPPO, HAPPO, MASAC, MADDPG) with minimal changes and demonstrates substantial gains in both convergence speed and data efficiency across VMAS and IsaacTeams benchmarks; ablations show that all components contribute and are interdependent, with pre-training and uncertainty modeling further enhancing performance. Overall, MAPO-LSO offers a scalable, generalizable enhancement to MARL by enriching the latent state space with dynamics-aware, jointly-consistent representations, potentially transforming practical efficiency in multi-agent domains.

Abstract

Sample efficiency remains a key challenge in multi-agent reinforcement learning (MARL). A promising approach is to learn a meaningful latent representation space through auxiliary learning objectives alongside the MARL objective to aid in learning a successful control policy. In our work, we present MAPO-LSO (Multi-Agent Policy Optimization with Latent Space Optimization) which applies a form of comprehensive representation learning devised to supplement MARL training. Specifically, MAPO-LSO proposes a multi-agent extension of transition dynamics reconstruction and self-predictive learning that constructs a latent state optimization scheme that can be trivially extended to current state-of-the-art MARL algorithms. Empirical results demonstrate MAPO-LSO to show notable improvements in sample efficiency and learning performance compared to its vanilla MARL counterpart without any additional MARL hyperparameter tuning on a diverse suite of MARL tasks.

Figures (13)

• Figure 1: A high-level illustration of the MAPO-LSO framework. For each agent $i = \{0,\dots,N\}$, the encoders ( ) embed their observations $o_t^i$ and propagates their encodings through a communication block ( ) that is subject to a communication network $\mathcal{G}(s_t)$. Once the agents communicate, the latent state $z_t^i$ ( ) is computed and used as inputs for its policy ( ) and value function ( ). For our MA-LSO procedure, the latent states are optimized using MA-Transition Dynamics Reconstruction (MA-TDR) and MA-Self-Predictive Learning (MA-SPL). These two learning processes are outlined in Section \ref{['ma-lso']} and loosely can be thought of as instilling the capability of inferring the observations and the next latent states of all agents from the current latent state.
• Figure 2: A detailed visualization of the MA-TDR modeling procedure with the auxiliary modules used to reconstruct transition dynamics for recurrent modeling and MA-PRL.
• Figure 3: The three MA-SPL subprocesses of MA-MR, MA-FDM and MA-IDM are shown.
• Figure 4: The graphs compare the collective returns under a normalized scale between various components introduced in this work --- namely, MAPO-LSO, phasic regularization, and uncertainty modeling (U.M.) --- over all VMAS and IST tasks and MARL algorithms, except for Figure \ref{['normalized-phasic']}, which normalizes over HAPPO, MADDPG and MASAC. The error bars indicate $\pm1$ std deviations. The results for the individual runs of all experiments are provided in Appendix \ref{['fullefficacy']}, \ref{['fullphasic']} and \ref{['fullnum']} respectively.
• Figure 5: MAPO-LSO as a pre-training process is evaluated, normalized on all runs listed in Appendix \ref{['fullpt']} with the error bars showing the $\pm1$ std deviation.

Theorems & Definitions (1)

• Definition 1