IMAGINE: Intelligent Multi-Agent Godot-based Indoor Networked Exploration
Tiago Leite, Maria Conceição, António Grilo
TL;DR
This work tackles autonomous, GNSS-denied indoor exploration by UAV swarms using MARL under realistic communication constraints. It presents a Godot-based, high-fidelity simulation with continuous action spaces and an ND-POMDP formulation, where agents share LiDAR and local maps to build coordinated strategies. A reward focused on new area discovery, a belief-based Local Map, and curriculum learning enable scalable training across seven increasingly complex levels, with CTDE often providing the best coordination in harder environments. Key findings include the importance of action/observation design, the benefit of LiDAR-specific perceptual architectures, and a robust, open simulation framework for deploying learned cooperation in physical robots, while noting MARL sample complexity and generalization challenges as areas for future work. $R(s,a,s') = W_{area} \cdot \Delta Area(s,s')$ with $\Delta Area(s,s') = (Area(s')-Area(s))/A_{\max}$ and $A_{\max} = 2 r v_{\max} \Delta t$ capture the core reward dynamics driving exploration.
Abstract
The exploration of unknown, Global Navigation Satellite System (GNSS) denied environments by an autonomous communication-aware and collaborative group of Unmanned Aerial Vehicles (UAVs) presents significant challenges in coordination, perception, and decentralized decision-making. This paper implements Multi-Agent Reinforcement Learning (MARL) to address these challenges in a 2D indoor environment, using high-fidelity game-engine simulations (Godot) and continuous action spaces. Policy training aims to achieve emergent collaborative behaviours and decision-making under uncertainty using Network-Distributed Partially Observable Markov Decision Processes (ND-POMDPs). Each UAV is equipped with a Light Detection and Ranging (LiDAR) sensor and can share data (sensor measurements and a local occupancy map) with neighbouring agents. Inter-agent communication constraints include limited range, bandwidth and latency. Extensive ablation studies evaluated MARL training paradigms, reward function, communication system, neural network (NN) architecture, memory mechanisms, and POMDP formulations. This work jointly addresses several key limitations in prior research, namely reliance on discrete actions, single-agent or centralized formulations, assumptions of a priori knowledge and permanent connectivity, inability to handle dynamic obstacles, short planning horizons and architectural complexity in Recurrent NNs/Transformers. Results show that the scalable training paradigm, combined with a simplified architecture, enables rapid autonomous exploration of an indoor area. The implementation of Curriculum-Learning (five increasingly complex levels) also enabled faster, more robust training. This combination of high-fidelity simulation, MARL formulation, and computational efficiency establishes a strong foundation for deploying learned cooperative strategies in physical robotic systems.