Collaborative Reinforcement Learning Based Unmanned Aerial Vehicle (UAV) Trajectory Design for 3D UAV Tracking

TL;DR

The paper addresses real-time 3D UAV tracking with one active and four passive UAVs by jointly optimizing the active UAV's transmit power and the trajectories of all controlled UAVs to minimize positioning error. It introduces ZD-RL, a distributional reinforcement learning framework that learns the probability distribution of the sum of future rewards through individual and global Z functions, enabling coordinated power and trajectory decisions. The authors provide theoretical results linking UAV geometry to localization accuracy (minimum error when passive-UAV distances are equal to $L_{\min}$) and demonstrate substantial gains (up to 39.4%–64.6% reductions in positioning error) over VD-RL and independent DRL baselines in various scenarios, along with convergence proofs and complexity analyses. The work offers a scalable, distributed approach to UAV-enabled 3D localization with practical implications for dynamic environments and energy-constrained deployments.

Abstract

In this paper, the problem of using one active unmanned aerial vehicle (UAV) and four passive UAVs to localize a 3D target UAV in real time is investigated. In the considered model, each passive UAV receives reflection signals from the target UAV, which are initially transmitted by the active UAV. The received reflection signals allow each passive UAV to estimate the signal transmission distance which will be transmitted to a base station (BS) for the estimation of the position of the target UAV. Due to the movement of the target UAV, each active/passive UAV must optimize its trajectory to continuously localize the target UAV. Meanwhile, since the accuracy of the distance estimation depends on the signal-to-noise ratio of the transmission signals, the active UAV must optimize its transmit power. This problem is formulated as an optimization problem whose goal is to jointly optimize the transmit power of the active UAV and trajectories of both active and passive UAVs so as to maximize the target UAV positioning accuracy. To solve this problem, a Z function decomposition based reinforcement learning (ZD-RL) method is proposed. Compared to value function decomposition based RL (VD-RL), the proposed method can find the probability distribution of the sum of future rewards to accurately estimate the expected value of the sum of future rewards thus finding better transmit power of the active UAV and trajectories for both active and passive UAVs and improving target UAV positioning accuracy. Simulation results show that the proposed ZD-RL method can reduce the positioning errors by up to 39.4% and 64.6%, compared to VD-RL and independent deep RL methods, respectively.

Figures (12)

• Figure 1: Illustration of the considered UAV localization network.
• Figure 2: The flow chart of the considered UAV positioning process.
• Figure 3: The flow chart of implementation.
• Figure 4: The actual trajectories of the target UAV and the estimated trajectories obtained by different methods.
• Figure 5: Value of the positioning error as the speed of the target UAV varies.