Scalable UAV Multi-Hop Networking via Multi-Agent Reinforcement Learning with Large Language Models

TL;DR

This work tackles the challenge of establishing scalable, reliable multi-hop UAV networks in disaster scenarios by marrying multi-agent reinforcement learning (MARL) with large language models (LLMs). The MRLMN framework introduces task-based agent grouping, reward decomposition, and behavioral constraints to improve scalability and robustness, and integrates an LLM-driven knowledge distillation module using Hungarian matching to guide MARL exploration. Empirical results in large-scale simulations show that MRLMN outperforms baseline MARL methods in terms of network coverage, data rate, and UAV availability, with ablations confirming the value of grouping, distillation, and constraints. The approach offers a practical route to quickly deploy resilient UAV-based communications in dynamic disaster environments, with potential extensions to energy management and real-world demonstrations.

Abstract

In disaster scenarios, establishing robust emergency communication networks is critical, and unmanned aerial vehicles (UAVs) offer a promising solution to rapidly restore connectivity. However, organizing UAVs to form multi-hop networks in large-scale dynamic environments presents significant challenges, including limitations in algorithmic scalability and the vast exploration space required for coordinated decision-making. To address these issues, we propose MRLMN, a novel framework that integrates multi-agent reinforcement learning (MARL) and large language models (LLMs) to jointly optimize UAV agents toward achieving optimal networking performance. The framework incorporates a grouping strategy with reward decomposition to enhance algorithmic scalability and balance decision-making across UAVs. In addition, behavioral constraints are applied to selected key UAVs to improve the robustness of the network. Furthermore, the framework integrates LLM agents, leveraging knowledge distillation to transfer their high-level decision-making capabilities to MARL agents. This enhances both the efficiency of exploration and the overall training process. In the distillation module, a Hungarian algorithm-based matching scheme is applied to align the decision outputs of the LLM and MARL agents and define the distillation loss. Extensive simulation results validate the effectiveness of our approach, demonstrating significant improvements in network performance, including enhanced coverage and communication quality.

Figures (8)

• Figure 1: In disaster scenarios, UAVs can rapidly establish temporary multi-hop wireless networks in communication dead zone, thereby restoring connectivity for UEs.
• Figure 2: UAVs are grouped by role, with each UAV deploying an independent PPO-based policy and critic network. Local observations are shared among agents, and each UAV receives individualized reward components.
• Figure 3: A knowledge distillation mechanism is proposed to transfer the LLM's decision-making capabilities to MARL agents. This design incorporates a bipartite matching strategy and a distillation loss function to align the decisions of the LLM and MARL agents, while simultaneously utilizing LLM's chain-of-thought reasoning.
• Figure 4: Training curves of different models over training steps, with the team reward computed according to equation \ref{['team_rew']}. Shaded regions represent the standard deviation.
• Figure 5: Comparison of model performance across three key performance metrics: connected UE proportion, average data rate, and available UAV ratio. The experiments are conducted under varying environment areas and the number of UAVs. Shaded regions represent the standard deviation.