Resilient UAV Trajectory Planning via Few-Shot Meta-Offline Reinforcement Learning

TL;DR

The paper tackles the challenge of safe, scalable UAV trajectory planning for wireless networks by addressing the limitations of online RL with offline data. It introduces a few-shot meta-offline RL framework that combines Conservative Q-Learning (CQL) with Model-Agnostic Meta-Learning (MAML) to learn robust initial policies from offline datasets and rapidly adapt to new environments with few data points. Empirical results show that the proposed CQL-MAML method converges faster and achieves near-optimal joint AoI and transmission-power performance, while exhibiting resilience to outages and environmental changes, outperforming baselines such as DQN and standard CQL. This work demonstrates the feasibility and benefits of integrating meta-learning with offline RL in wireless UAV applications, enabling safe, scalable, and adaptive decision-making for precision agriculture and environmental monitoring.

Abstract

Reinforcement learning (RL) has been a promising essence in future 5G-beyond and 6G systems. Its main advantage lies in its robust model-free decision-making in complex and large-dimension wireless environments. However, most existing RL frameworks rely on online interaction with the environment, which might not be feasible due to safety and cost concerns. Another problem with online RL is the lack of scalability of the designed algorithm with dynamic or new environments. This work proposes a novel, resilient, few-shot meta-offline RL algorithm combining offline RL using conservative Q-learning (CQL) and meta-learning using model-agnostic meta-learning (MAML). The proposed algorithm can train RL models using static offline datasets without any online interaction with the environments. In addition, with the aid of MAML, the proposed model can be scaled up to new unseen environments. We showcase the proposed algorithm for optimizing an unmanned aerial vehicle (UAV) 's trajectory and scheduling policy to minimize the age-of-information (AoI) and transmission power of limited-power devices. Numerical results show that the proposed few-shot meta-offline RL algorithm converges faster than baseline schemes, such as deep Q-networks and CQL. In addition, it is the only algorithm that can achieve optimal joint AoI and transmission power using an offline dataset with few shots of data points and is resilient to network failures due to unprecedented environmental changes.

Figures (9)

• Figure 1: Illustration of the system model. We consider smart agriculture, where a flying UAV collects information from ground nodes. In addition, sudden heavy rain occurs, which affects communication. The objective is to jointly minimize the AoI and transmission power while considering dynamic and unpredictable sources in the environment.
• Figure 2: Illustration of Offline RL, which involves two phases: data collection and offline learning. In the data collection phase, fixed datasets are collected using behavioral policies. A learning model uses a static offline dataset to find the optimum policy in the offline learning phase.
• Figure 3: Illustration of the proposed CQL-MAML algorithm, composed of meta-RL training and testing phases. The former utilizes offline training, using the CQL algorithm, across different tasks (environments) with different objectives to find the optimum initial parameters. In contrast, the latter performs a few offline SGD steps over the reached weights on a new unseen task.
• Figure 4: An illustration of the meta-training performance of the proposed CQL-MAML algorithm using $8$ meta-tasks and an offline dataset with $500$ data points. Both loss and rewards (normalized by $100$ converge to their minimum and maximum values, respectively.
• Figure 5: The meta-testing convergence of the proposed CQL-MAML algorithm compared to baseline schemes in a new unseen task after training the initial weights using $8$ meta-tasks and an offline dataset with $500$ data points.