Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Power Allocation for Delay Optimization in Device-to-Device Networks: A Graph Reinforcement Learning Approach

Hao Fang, Kai Huang, Hao Ye, Chongtao Guo, Le Liang, Xiao Li, Shi Jin

TL;DR

The work tackles delay-aware power allocation in D2D networks by casting resource control as a graph-structured reinforcement learning problem. It proposes a centralized PPO agent with GNNs embedded in both the actor and critic, using a delay-centric state that includes channel information and buffer dynamics ${S=\{\boldsymbol{H}, \boldsymbol{T_D}, \boldsymbol{\Delta}, \boldsymbol{\Xi}\}\}$ and an objective $\min_{\mathbf{p}} \frac{1}{M}\sum_{i=1}^M \frac{1}{m(i)} \sum_{g=1}^{m(i)} T_{D_i}(g)$ under $0\le p_i \le P_{\max}$. The method introduces a delay-aware reward $R_n=-\sum_i J_i[n]$ and integrates GNN layers into both PPO networks to capture interference topology while enabling end-to-end gradient updates. Results show significant delay reductions and fairness improvements over baselines such as ITLinQ and WMMSE, with strong scalability and generalization to unseen network sizes, densities, and user distributions. The approach paves the way for practical, topology-aware power control in next-generation wireless networks, with potential extensions to distributed or partially observable GNN-RL architectures for reduced centralization.

Abstract

The pursuit of rate maximization in wireless communication frequently encounters substantial challenges associated with user fairness. This paper addresses these challenges by exploring a novel power allocation approach for delay optimization, utilizing graph neural networks (GNNs)-based reinforcement learning (RL) in device-to-device (D2D) communication. The proposed approach incorporates not only channel state information but also factors such as packet delay, the number of backlogged packets, and the number of transmitted packets into the components of the state information. We adopt a centralized RL method, where a central controller collects and processes the state information. The central controller functions as an agent trained using the proximal policy optimization (PPO) algorithm. To better utilize topology information in the communication network and enhance the generalization of the proposed method, we embed GNN layers into both the actor and critic networks of the PPO algorithm. This integration allows for efficient parameter updates of GNNs and enables the state information to be parameterized as a low-dimensional embedding, which is leveraged by the agent to optimize power allocation strategies. Simulation results demonstrate that the proposed method effectively reduces average delay while ensuring user fairness, outperforms baseline methods, and exhibits scalability and generalization capability.

Power Allocation for Delay Optimization in Device-to-Device Networks: A Graph Reinforcement Learning Approach

TL;DR

The work tackles delay-aware power allocation in D2D networks by casting resource control as a graph-structured reinforcement learning problem. It proposes a centralized PPO agent with GNNs embedded in both the actor and critic, using a delay-centric state that includes channel information and buffer dynamics and an objective under . The method introduces a delay-aware reward and integrates GNN layers into both PPO networks to capture interference topology while enabling end-to-end gradient updates. Results show significant delay reductions and fairness improvements over baselines such as ITLinQ and WMMSE, with strong scalability and generalization to unseen network sizes, densities, and user distributions. The approach paves the way for practical, topology-aware power control in next-generation wireless networks, with potential extensions to distributed or partially observable GNN-RL architectures for reduced centralization.

Abstract

The pursuit of rate maximization in wireless communication frequently encounters substantial challenges associated with user fairness. This paper addresses these challenges by exploring a novel power allocation approach for delay optimization, utilizing graph neural networks (GNNs)-based reinforcement learning (RL) in device-to-device (D2D) communication. The proposed approach incorporates not only channel state information but also factors such as packet delay, the number of backlogged packets, and the number of transmitted packets into the components of the state information. We adopt a centralized RL method, where a central controller collects and processes the state information. The central controller functions as an agent trained using the proximal policy optimization (PPO) algorithm. To better utilize topology information in the communication network and enhance the generalization of the proposed method, we embed GNN layers into both the actor and critic networks of the PPO algorithm. This integration allows for efficient parameter updates of GNNs and enables the state information to be parameterized as a low-dimensional embedding, which is leveraged by the agent to optimize power allocation strategies. Simulation results demonstrate that the proposed method effectively reduces average delay while ensuring user fairness, outperforms baseline methods, and exhibits scalability and generalization capability.
Paper Structure (21 sections, 2 theorems, 29 equations, 11 figures, 5 tables, 1 algorithm)

This paper contains 21 sections, 2 theorems, 29 equations, 11 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Statement and Motivation
  3. Related Work
  4. Contribution
  5. Paper Organization
  6. System Model and Problem Formulation
  7. GNN-Based RL Method
  8. State
  9. Action Space
  10. Reward Design
  11. Learning Algorithm
  12. Simulation Results
  13. Wireless Network and Parameters Settings
  14. Baseline Methods
  15. Convergence and Validity Analysis
  16. ...and 6 more sections

Key Result

Proposition 1

Given that the number of D2D pairs, $M$, and the average packet arrival rate, $\lambda$, are constant, maximizing the expected return in the MDP framework becomes asymptotically equivalent to minimizing the average delay across all D2D pairs in the original problem. This equivalence can be expressed where $\left( \cdot \right)$$\Leftrightarrow$$\left( \cdot \right)$ denotes that the former is equi

Figures (11)

  • Figure 1: Illustration of a D2D communication scenario.
  • Figure 2: The overall process of the proposed method.
  • Figure 3: Illustration of implementing GNN architecture with five D2D pairs.
  • Figure 4: The impact of the number of GNN layers.
  • Figure 5: Return for each training episode with increasing iterations.
  • ...and 6 more figures

Theorems & Definitions (2)

  • Proposition 1
  • Proposition 2