User Association and Coordinated Beamforming in Cognitive Aerial-Terrestrial Networks: A Safe Reinforcement Learning Approach

TL;DR

This work tackles spectrum sharing in cognitive aerial-terrestrial networks by jointly optimizing user association and coordinated beamforming to maximize terrestrial throughput without violating aerial interference caps. It introduces a safe, multi-agent reinforcement learning framework that models the problem as a networked constrained partially observable Markov game (NCPOMG), with UAVs as primary users and TUs as secondary users. BSs (beamformers) learn safe policies via Constrained Update Projection (CUP) while UAs (TUs) learn UA policies through a D3QN framework, all within a decentralized, DTDE setting that relies on compressed, non-global observations to reduce CSI requirements. Simulation results show the proposed safe DRL approach outperforms two-stage optimization and penalty-based DRL schemes in sum-rate performance while maintaining AU interference below the threshold, and it does so with lower computational and communication overhead. This indicates practical applicability for dynamic CATNs where real-time CSI is costly and safety constraints are critical.

Abstract

Cognitive aerial-terrestrial networks (CATNs) offer a solution to spectrum scarcity by sharing spectrum between aerial and terrestrial networks. However, aerial users (AUs) experience significant interference from numerous terrestrial base stations (BSs). To alleviate such interference, we investigate a user association and coordinated beamforming (CBF) problem in CATN, where the aerial network serves as the primary network sharing its spectrum with the terrestrial network. Specifically, we maximize the sum rate of the secondary terrestrial users (TUs) under the interference temperature constraints of the AUs. Traditional iterative optimization schemes are impractical due to their high computational complexity and information exchange overhead. Although deep reinforcement learning (DRL) based schemes can address these challenges, their performance is sensitive to the weights of the weighted penalty terms for violating constraints in the reward function. Motivated by these issues, we propose a safe DRL-based user association and CBF scheme for CATN, eliminating the need for training multiple times to find the optimal penalty weight before actual deployment. Specifically, the CATN is modeled as a networked constrained partially observable Markov game. Each TU acts as an agent to choose its associated BS, and each BS acts as an agent to decide its beamforming vectors, aiming to maximize the reward while satisfying the safety constraints introduced by the interference constraints of the AUs. By exploiting a safe DRL algorithm, the proposed scheme incurs lower deployment expenses than the penalty-based DRL schemes since only one training is required before actual deployment. Simulation results show that the proposed scheme can achieve a higher sum rate of TUs than a two-stage optimization scheme while the average received interference power of the AUs is generally below the threshold.

Figures (8)

• Figure 1: Cognitive aerial-terrestrial network.
• Figure 2: Illustration for the workflow of the proposed safe DRL-based DDUACBF framework.
• Figure 3: Simulation scenario.
• Figure 4: Performance comparison of the proposed scheme and the benchmarks: $P_{\max}=20{\rm{W}}$ and ${I_{\max}} = 1.6\times {10^{ - 10}}{\rm{mW}}$.
• Figure 5: Performance of the penalty-based PPO with different penalty weights $\zeta$: $P_{\max}=20{\rm{W}}$ and ${I_{\max}} = 1.6\times {10^{ - 10}}{\rm{mW}}$.