Two-Layer Reinforcement Learning-Assisted Joint Beamforming and Trajectory Optimization for Multi-UAV Downlink Communications

TL;DR

This work tackles the challenge of jointly optimizing beamforming and UAV trajectories in a dynamic multi-UAV downlink by introducing a timescale-separated framework. The inner loop deploys a topology-aware graph neural network with GraphNorm to achieve sub-millisecond beamforming inference, while the outer loop uses centralized-training, decentralized-execution MAPPO to learn cooperative trajectory policies under a CTDE paradigm. A dynamic, time-varying heterogeneous graph models the changing UAV-user associations, and a reachability-aware reward with arrival masking stabilizes long-horizon learning. Results demonstrate strong generalization to varying network densities, superior long-term throughput, and real-time inference, underscoring the practical potential for scalable 6G non-terrestrial networks.

Abstract

Unmanned aerial vehicles (UAVs) are pivotal for future 6G non-terrestrial networks, yet their high mobility creates a complex coupled optimization problem for beamforming and trajectory design. Existing numerical methods suffer from prohibitive latency, while standard deep learning often ignores dynamic interference topology, limiting scalability. To address these issues, this paper proposes a hierarchically decoupled framework synergizing graph neural networks (GNNs) with multi-agent reinforcement learning. Specifically, on the short timescale, we develop a topology-aware GNN beamformer by incorporating GraphNorm. By modeling the dynamic UAV-user association as a time-varying heterogeneous graph, this method explicitly extracts interference patterns to achieve sub-millisecond inference. On the long timescale, trajectory planning is modeled as a decentralized partially observable Markov decision process and solved via the multi-agent proximal policy optimization algorithm under the centralized training with decentralized execution paradigm, facilitating cooperative behaviors. Extensive simulation results demonstrate that the proposed framework significantly outperforms state-of-the-art optimization heuristics and deep learning baselines in terms of system sum rate, convergence speed, and generalization capability.

Figures (10)

• Figure 1: System model
• Figure 2: Overall framework of the proposed GNN-enabled beamforming and MAPPO-based UAV trajectory optimization.
• Figure 3: Average sum rate on training and validation datasets across epochs.
• Figure 4: Sum rate versus UAV maximum transmission power.
• Figure 5: Sum rate vs. noise power.