Intelligent Spectrum Sharing in Integrated TN-NTNs: A Hierarchical Deep Reinforcement Learning Approach

TL;DR

The paper addresses spectrum sharing in integrated TN-NTNs, where multi-tier, heterogeneous networks create a high-dimensional, dynamic interference landscape. It introduces a hierarchical deep reinforcement learning (HDRL) framework with global, regional, and local policies that coordinate spectrum allocation across timescales, using proximal policy optimization (PPO) for stable learning. Key contributions include a three-tier system model (LEO, HAPs, UAVs, TBSs), tier-specific reward functions, a learning process that propagates subgoals across tiers, and a detailed complexity analysis demonstrating scalability benefits. Simulation results show HDRL attaining near-optimal spectral efficiency and higher throughput with favorable execution times compared to exhaustive search, random access, PPO, and MAPPO baselines, highlighting its practical potential for future 6G networks.

Abstract

Integrating non-terrestrial networks (NTNs) with terrestrial networks (TNs) is key to enhancing coverage, capacity, and reliability in future wireless communications. However, the multi-tier, heterogeneous architecture of these integrated TN-NTNs introduces complex challenges in spectrum sharing and interference management. Conventional optimization approaches struggle to handle the high-dimensional decision space and dynamic nature of these networks. This paper proposes a novel hierarchical deep reinforcement learning (HDRL) framework to address these challenges and enable intelligent spectrum sharing. The proposed framework leverages the inherent hierarchy of the network, with separate policies for each tier, to learn and optimize spectrum allocation decisions at different timescales and levels of abstraction. By decomposing the complex spectrum sharing problem into manageable sub-tasks and allowing for efficient coordination among the tiers, the HDRL approach offers a scalable and adaptive solution for spectrum management in future TN-NTNs. Simulation results demonstrate the superior performance of the proposed framework compared to traditional approaches, highlighting its potential to enhance spectral efficiency and network capacity in dynamic, multi-tier environments.

Figures (5)

• Figure 1: Illustration of the hierarchical policy structure and agent-environment interaction loop.
• Figure 2: System model and HDRL framework for spectrum sharing in integrated TN-NTNs.
• Figure 3: Normalized average cumulative reward of the proposed HDRL framework for different network hierarchies.
• Figure 4: Spectral efficiency achieved by different algorithms for different network hierarchies.
• Figure 5: Average network throughput achieved by different algorithms.