Breaking the Pre-Planning Barrier: Adaptive Real-Time Coordination of Heterogeneous UAVs

TL;DR

This work addresses the challenge of rigid, pre-planned UAV routing in dynamic environments by introducing HGAM, a heterogeneous graph attention based multi-agent reinforcement learning framework. HGAM combines a heterogeneous graph representation with a continuous-action actor-critic architecture and a centralized critic to enable real-time coordination between data-collecting MUAVs and charging CUAVs under partial observability. Key contributions include the CTDE-enabled HGAM, a tailored graph encoder/attention/execution pipeline, and training methodologies such as dilemma detection, N-step returns, and prioritized experience replay. The approach yields substantial improvements in data collection coverage and charging efficiency, demonstrating a practical pathway toward scalable, autonomous, and robust UAV networks in complex, uncertain environments.

Abstract

Unmanned Aerial Vehicles (UAVs) offer significant potential in dynamic, perception-intensive tasks such as search and rescue and environmental monitoring; however, their effectiveness is severely restricted by conventional pre-planned routing methods, which lack the flexibility to respond in real-time to evolving task demands, unexpected disturbances, and localized view limitations in real-world scenarios. To address this fundamental limitation, we introduce a novel multi-agent reinforcement learning framework named \textbf{H}eterogeneous \textbf{G}raph \textbf{A}ttention \textbf{M}ulti-agent Deep Deterministic Policy Gradient (HGAM), uniquely designed to enable adaptive real-time coordination between mission UAVs (MUAVs) and charging UAVs (CUAVs). HGAM specifically addresses the previously unsolved challenge of enabling precise, decentralized continuous-action coordination solely based on local, heterogeneous graph-based observations. Extensive simulations demonstrate that HGAM substantially surpasses existing methods, achieving, for example, a 30\% improvement in data collection coverage and a 20\% increase in charging efficiency, providing crucial insights and foundations for the future deployment of intelligent, flexible UAV networks in complex, dynamic environments.

Figures (7)

• Figure 1: Illustration of adaptive real-time coordination between three MUAVs and a CUAV in a dynamic urban environment. MUAVs autonomously sense and collect data from PoIs, depicted within their sensing range (yellow dashed circles), while CUAV proactively delivers wireless charging to MUAVs in need, indicated by the charging range (green dashed circles). UAV communication (red dashed lines) enables decentralized coordination and obstacle avoidance under limited local observations.
• Figure 2: Illustration of MUAV sensing and CUAV charging ranges($d_t^v$ and $l_t^p$), highlighting collaborative interactions with PoIs.
• Figure 3: Overview of the actor-critic architecture with heterogeneous GAT. The encoder and GAT module collaboratively generate node embeddings. The actor network ($\pi$ Layer) utilizes local embeddings for decentralized real-time decisions, while the critic network (Q Layer) applies global embeddings for centralized evaluation of joint state-action values, enhancing multi-agent cooperation.
• Figure 4: Overall HGAM pipeline under the CTDE paradigm. Actor networks utilize local graph embeddings for decentralized, real-time decisions, while the critic network employs global graph embeddings for centralized Q-value estimation during training. Experiences collected in the Prioritized Experience Replay (PER) buffer are prioritized based on TD errors, enhancing training stability and performance.
• Figure 5: Adaptive UAV trajectories generated by HGAM under local-view constraints. Yellow stars indicate initial positions. (a) MUAV paths (purple and red), demonstrating efficient and complementary coverage. (b) CUAV trajectory (yellow) dynamically supports MUAVs via adaptive charging, while avoiding obstacles.