Investigating Relational State Abstraction in Collaborative MARL

TL;DR

This work tackles sample efficiency in collaborative MARL by introducing MARC, a simple yet effective relational critic that encodes observations as a spatial graph and processes them with a relational graph neural network. By exploiting translation-invariant spatial relations and a shared observation encoder within a centralized training/decentralized execution framework, MARC improves both sample efficiency and asymptotic performance across six spatially demanding tasks, including heterogeneous agents and a continuous-domain task. Extensive ablations show the default spatial relation set provides strong inductive bias, while overly fine-grained or fully dense graphs can hinder efficiency; MARC also demonstrates robust generalization to varying agent and object configurations. The findings highlight relational state abstraction as a practical avenue for more efficient MARL in spatially complex environments, with potential extensions to richer inductive biases and interpretability for real-world deployment.

Abstract

This paper explores the impact of relational state abstraction on sample efficiency and performance in collaborative Multi-Agent Reinforcement Learning. The proposed abstraction is based on spatial relationships in environments where direct communication between agents is not allowed, leveraging the ubiquity of spatial reasoning in real-world multi-agent scenarios. We introduce MARC (Multi-Agent Relational Critic), a simple yet effective critic architecture incorporating spatial relational inductive biases by transforming the state into a spatial graph and processing it through a relational graph neural network. The performance of MARC is evaluated across six collaborative tasks, including a novel environment with heterogeneous agents. We conduct a comprehensive empirical analysis, comparing MARC against state-of-the-art MARL baselines, demonstrating improvements in both sample efficiency and asymptotic performance, as well as its potential for generalization. Our findings suggest that a minimal integration of spatial relational inductive biases as abstraction can yield substantial benefits without requiring complex designs or task-specific engineering. This work provides insights into the potential of relational state abstraction to address sample efficiency, a key challenge in MARL, offering a promising direction for developing more efficient algorithms in spatially complex environments.

Figures (12)

• Figure 1: Overview of our MARC architecture on the example of level-based foraging. Without adding information, the observation is constructed as a graph, with objects and agents as entities and our chosen set of relations. We then pass this relational graph into a shared R-GCN component, followed by an individual head for each agent to estimate the state-action value.
• Figure 2: Mean average performance and $95\%$ confidence interval for all discrete tasks. For each model, we run 3 random seeds.
• Figure 3: Collaborative pick and place environment on a 5x5 grid with 1 picker agent, 1 delivery agent, 3 boxes and 3 goal locations.
• Figure 4: Level-based Foraging environment with 4 agents and fruits that become trees once picked.
• Figure 5: Wolfpack environment with 3 predator agents coordinating to catch 2 moving prey targets.