Digital-Twin Empowered Deep Reinforcement Learning For Site-Specific Radio Resource Management in NextG Wireless Aerial Corridor

TL;DR

The paper proposes a Digital Twin–driven two-stage framework to solve joint UAV-BS association and beam selection in NextG aerial corridors, addressing high-dimensional action spaces, CSI overhead, and stringent latency. It combines a physics-based Channel Twin obtained from high-fidelity ray tracing with a Multi-Head PPO policy that maps CT features to joint UAV-BS-beam decisions, using a shared backbone and per-UAV actor heads to scale with UAV density. The DT-assisted training pipeline yields strong throughput gains and millisecond-level inference, outperforming optimization-based and learning-based baselines while maintaining near-real-time operation in a site-specific 3D environment. The work demonstrates the practicality and effectiveness of DTs for offline DRL training and real-time resource management in autonomous aerial networks, with potential extensions to mobility-aware and multi-agent coordination. The key contributions include the DT-driven beam-gain Stage 1 optimization, the MH-PPO Stage 2 policy, and a load-balancing reward design that enforces capacity constraints without hardcoding, delivering robust performance in unseen channel realizations.

Abstract

Joint base station (BS) association and beam selection in multi-UAV aerial corridors constitutes a challenging radio resource management (RRM) problem. It is driven by high-dimensional action spaces, need for substantial overhead to acquire global channel state information (CSI), rapidly varying propagation channels, and stringent latency requirements. Conventional combinatorial optimization methods, while near-optimal, are computationally prohibitive for real-time operation in such dynamic environments. While learning-based approaches can mitigate computational complexity and CSI overhead, the need for extensive site-specific (SS) datasets for model training remains a key challenge. To address these challenges, we develop a Digital Twin (DT)-enabled two-stage optimization framework that couples physics-based beam gain modeling with DRL for scalable online decision-making. In the first stage, a channel twin (CT) is constructed using a high-fidelity ray-tracing solver with geo-spatial contexts, and network information to capture SS propagation characteristics, and dual annealing algorithm is employed to precompute optimal transmission beam directions. In the second stage, a Multi-Head Proximal Policy Optimization (MH-PPO) agent, equipped with a scalable multi-head actor-critic architecture, is trained on the DT-generated channel dataset to directly map complex channel and beam states to jointly execute UAV-BS-beam association decisions. The proposed PPO agent achieves a 44%-121% improvement over DQN and 249%-807% gain over traditional heuristic based optimization schemes in a dense UAV scenario, while reducing inference latency by several orders of magnitude. These results demonstrate that DT-driven training pipelines can deliver high-performance, low-latency RRM policies tailored to SS deployments suitable for real-time resource management in next-generation aerial corridor networks.

Figures (10)

• Figure 1: UAVs and BSs in the site-specific (SS) DT environment.
• Figure 2: Workflow of the proposed framework.
• Figure 3: Proposed Multi-Head PPO based UAV-BS-Beam Association Architecture
• Figure 4: Altitude vs. per-UAV throughput, $M$ = 20
• Figure 5: Altitude vs. per-UAV throughput, $M$ = 25