Graph Based Deep Reinforcement Learning Aided by Transformers for Multi-Agent Cooperation

TL;DR

This work tackles cooperative multi-agent drone navigation under partial observability and limited communication by proposing a graph-based framework that couples a GNN with transformer-based message passing to capture spatial and long-range dependencies, and a Double DQN with prioritized experience replay for robust policy learning. Key innovations include adaptive graph construction, edge-feature enhanced attention, and an entity-aware embedding scheme that feed a DDQN to produce coordinated action choices. Empirical results in a MiniGrid-like grid environment show the approach achieves around 90% goal collection and near-complete grid coverage while reducing episode steps to roughly 200, substantially outperforming PSO, DBSCAN, greedy, and standard DQN baselines. The framework demonstrates improved scalability and adaptability for real-world, communication-constrained multi-agent missions with partial observability, highlighting the practical impact for disaster response and environmental monitoring tasks.

Abstract

Mission planning for a fleet of cooperative autonomous drones in applications that involve serving distributed target points, such as disaster response, environmental monitoring, and surveillance, is challenging, especially under partial observability, limited communication range, and uncertain environments. Traditional path-planning algorithms struggle in these scenarios, particularly when prior information is not available. To address these challenges, we propose a novel framework that integrates Graph Neural Networks (GNNs), Deep Reinforcement Learning (DRL), and transformer-based mechanisms for enhanced multi-agent coordination and collective task execution. Our approach leverages GNNs to model agent-agent and agent-goal interactions through adaptive graph construction, enabling efficient information aggregation and decision-making under constrained communication. A transformer-based message-passing mechanism, augmented with edge-feature-enhanced attention, captures complex interaction patterns, while a Double Deep Q-Network (Double DQN) with prioritized experience replay optimizes agent policies in partially observable environments. This integration is carefully designed to address specific requirements of multi-agent navigation, such as scalability, adaptability, and efficient task execution. Experimental results demonstrate superior performance, with 90% service provisioning and 100% grid coverage (node discovery), while reducing the average steps per episode to 200, compared to 600 for benchmark methods such as particle swarm optimization (PSO), greedy algorithms and DQN.

Figures (7)

• Figure 1: Example configuration of the grid environment with 5 agents and 20 goals, uncollected goals (green), collected goals (grey), and agent vision radius $r_v$ (red circle around agents). Drones are allowed to communicate with at most $k=3$ adjacent drones.
• Figure 2: Proposed GNN architecture consisting of an entity embedding layer and transformer-based message passing mechanism.
• Figure 3: The heatmap of attention weights generated by the transformer-based architecture. Rows represent source nodes (agents), and columns represent target nodes (goals). Brighter colors (higher weights) indicate stronger attention, highlighting critical agent-goal relationships.
• Figure 4: Comparison between the proposed method and the baseline DQN method in terms of (a) goal achievement and (b) grid coverage.
• Figure 5: Performance comparison between GNN and DQN approaches showing the percentage of goals collected over time with 15 agents.