Age and Power Minimization via Meta-Deep Reinforcement Learning in UAV Networks

TL;DR

The paper tackles joint minimization of AoI and transmission power in UAV-enabled IoT networks by formulating a multi-objective optimization and solving it with a meta-deep RL framework. It integrates a Deep Q-Network (DQN) with Model-Agnostic Meta-Learning (MAML) to learn a fast-adapting initialization that can handle varying trade-offs parameterized by $\lambda$ with only a few training samples. The proposed method demonstrates faster convergence and better combined AoI/power performance than baseline deep RL across multiple task settings, highlighting improved adaptability to changing network configurations. This approach enables scalable, rapid deployment of UAV data-collection policies in dynamic wireless environments, with potential extensions to multi-UAV collaborations and real-time reconfiguration.

Abstract

Age-of-information (AoI) and transmission power are crucial performance metrics in low energy wireless networks, where information freshness is of paramount importance. This study examines a power-limited internet of things (IoT) network supported by a flying unmanned aerial vehicle(UAV) that collects data. Our aim is to optimize the UAV flight trajectory and scheduling policy to minimize a varying AoI and transmission power combination. To tackle this variation, this paper proposes a meta-deep reinforcement learning (RL) approach that integrates deep Q-networks (DQNs) with model-agnostic meta-learning (MAML). DQNs determine optimal UAV decisions, while MAML enables scalability across varying objective functions. Numerical results indicate that the proposed algorithm converges faster and adapts to new objectives more effectively than traditional deep RL methods, achieving minimal AoI and transmission power overall.

Figures (6)

• Figure 1: The considered system model operating within a grid environment, consists of a UAV that serves multiple, fixed located IoT devices with limited power.
• Figure 2: A visualization of the proposed meta-RL algorithm using MAML and DQNs. The left block represents the interaction between the DQN agent and the environment, where state transitions are stored in the replay buffer for training. The shared initial weights ($\theta$) derived from the meta-learned initialization, are used for all tasks and fine-tuned through task-specific gradient updates, resulting in task-specific weights ($\theta^{'}_1$,$\theta^{'}_2$,$\theta^{'}_3$). The updated weights are used to calculate individual task losses, which are then aggregated into a meta-loss to compute the meta-update ($\theta^{0}$). We utilize learning across different tasks trained using DQN agents to reach initial weights that can adapt quickly to new unseen tasks.
• Figure 3: Meta-training performance: (a) the convergence of the meta-losses, (b) the convergence of the rewards as a function of meta-training epochs tested over $11$ different test environments, each with $5$ devices, and (c) the convergence of the rewards as a function of meta-training epochs tested over $11$ different test environments, each with $10$ devices.
• Figure 4: Meta-testing convergence compared to random weights initialization with an environment with $\lambda = 300$.
• Figure 5: Achieved rewards after $30$ episodes tested over $11$ different test environments.