On Designing Multi-UAV aided Wireless Powered Dynamic Communication via Hierarchical Deep Reinforcement Learning

TL;DR

This work tackles the challenge of designing a dynamic, energy-aware UAV-aided WPCN where wireless nodes can repeatedly switch between energy harvesting and data transmission. It introduces a two-tier MAHDRL framework, using centralized SAC for continuous trajectory and WET decisions (tier-1) and distributed DQN for discrete WDC scheduling (tier-2), with partial observability modeled as a POMDP. A double-threshold WN updating rule and non-linear energy harvesting are integrated to reflect time-varying energy demands, while the LoS-based A2G channel model and energy constraints shape the optimization. Simulation results show substantial performance gains over state-of-the-art benchmarks, demonstrating improved total transmitted data, scalability with network size, and robust handling of dynamic WN behavior, making the approach promising for practical UAV-enabled WPCNs.

Abstract

This paper proposes a novel design on the wireless powered communication network (WPCN) in dynamic environments under the assistance of multiple unmanned aerial vehicles (UAVs). Unlike the existing studies, where the low-power wireless nodes (WNs) often conform to the coherent harvest-then-transmit protocol, under our newly proposed double-threshold based WN type updating rule, each WN can dynamically and repeatedly update its WN type as an E-node for non-linear energy harvesting over time slots or an I-node for transmitting data over sub-slots. To maximize the total transmission data size of all the WNs over T slots, each of the UAVs individually determines its trajectory and binary wireless energy transmission (WET) decisions over times slots and its binary wireless data collection (WDC) decisions over sub-slots, under the constraints of each UAV's limited on-board energy and each WN's node type updating rule. However, due to the UAVs' tightly-coupled trajectories with their WET and WDC decisions, as well as each WN's time-varying battery energy, this problem is difficult to solve optimally. We then propose a new multi-agent based hierarchical deep reinforcement learning (MAHDRL) framework with two tiers to solve the problem efficiently, where the soft actor critic (SAC) policy is designed in tier-1 to determine each UAV's continuous trajectory and binary WET decision over time slots, and the deep-Q learning (DQN) policy is designed in tier-2 to determine each UAV's binary WDC decisions over sub-slots under the given UAV trajectory from tier-1. Both of the SAC policy and the DQN policy are executed distributively at each UAV. Finally, extensive simulation results are provided to validate the outweighed performance of the proposed MAHDRL approach over various state-of-the-art benchmarks.

Figures (10)

• Figure 1: Multi-UAV aided WPCN with repeatedly changing WN types.
• Figure 2: Central training of the SAC policy.
• Figure 3: Local training of the DQN at UAV-$u$.
• Figure 4: Comparison of MAHDRL with DDPG+DQN.
• Figure 5: Comparison of MAHDRL with benchmarks.