A Deep Q-Learning based, Base-Station Connectivity-Aware, Decentralized Pheromone Mobility Model for Autonomous UAV Networks

TL;DR

A neighborhood- and BS-connectivity-aware distributed pheromone mobility model to autonomously coordinate the UAV movements in a decentralized network is developed and extended to a deep $Q$-learning policy variant to further tune and improve the balance between coverage and connectivity.

Abstract

UAV networks consisting of low SWaP (size, weight, and power), fixed-wing UAVs are used in many applications, including area monitoring, search and rescue, surveillance, and tracking. Performing these operations efficiently requires a scalable, decentralized, autonomous UAV network architecture with high network connectivity. Whereas fast area coverage is needed for quickly sensing the area, strong node degree and base station (BS) connectivity are needed for UAV control and coordination and for transmitting sensed information to the BS in real time. However, the area coverage and connectivity exhibit a fundamental trade-off: maintaining connectivity restricts the UAVs' ability to explore. In this paper, we first present a node degree and BS connectivity-aware distributed pheromone (BS-CAP) mobility model to autonomously coordinate the UAV movements in a decentralized UAV network. This model maintains a desired connectivity among 1-hop neighbors and to the BS while achieving fast area coverage. Next, we propose a deep Q-learning policy based BS-CAP model (BSCAP-DQN) to further tune and improve the coverage and connectivity trade-off. Since it is not practical to know the complete topology of such a network in real time, the proposed mobility models work online, are fully distributed, and rely on neighborhood information. Our simulations demonstrate that both proposed models achieve efficient area coverage and desired node degree and BS connectivity, improving significantly over existing schemes.

Figures (9)

• Figure 1: Decentralized, autonomous UAV network performing monitoring, search and surveillance in inaccessible disaster areas.
• Figure 2: UAV headings and next-waypoints. (a) Given its current heading, a UAV selects one (green) of the five forward-facing cells out of its eight possible next-waypoint cells (gray). (b) Arriving at this waypoint, the UAV selects its new next-waypoint; discretizing its current heading, waypoint 2 is the closest to straight, so that its forward-facing options are $\{0,1,2,3,4\}$.
• Figure 3: We use offline pre-training, followed by online training, to improve training efficiency while accounting for the effect of policy changes in our multi-UAV setting CAPDQN_ref.
• Figure 4: Coverage vs. Time plots for 30 and 50 UAVs at 20 m/s and 40 m/s. While more coverage in less time is preferred, we experiment with several parameter settings to balance coverage with connectivity (illustrated in subsequent plots); the parameter values are chosen to produce similar coverage curves among the three tested algorithms (BS-CAP, BSCAP-DQN, and ConCov). The intermediate parameter settings ($\beta = 1.5$, $n=3$, and $\omega=0.3$) are omitted from these plots for clarity.
• Figure 5: Coverage Fairness performance plots for 30 and 50 UAVs, at 20 m/s and 40 m/s. (a) For 30 UAVs, ConCov achieves higher $F$ values than BS-CAP and BSCAP-DQN at lower $T_c$ values. The $F$ values of all the models converge when $T_c$ increases. (b) For 50 UAVs, BSCAP-DQN achieves higher $F$ values than BS-CAP and ConCov at lower $T_c$ values.