Improve the Training Efficiency of DRL for Wireless Communication Resource Allocation: The Role of Generative Diffusion Models

TL;DR

This work addresses the training-efficiency bottlenecks of DRL for dynamic wireless resource allocation by introducing Diffusion-based Deep Reinforcement Learning (D2RL). D2RL leverages Generative Diffusion Models to enhance state sampling, action exploration, and reward design, with two modes (Mode I for reward-focused exploration and Mode II for state-focused exploration) to adapt to available data. The authors provide gradient-analytic insights and implement specialized exploration networks (SENPNN, AENPNN, RENPNN) that integrate into a DRL framework, demonstrated on a full-duplex wireless system where D2RL accelerates convergence and reduces total training cost while maintaining competitive performance. The findings highlight the practical potential of incorporating GDMs into DRL pipelines for real-time wireless deployments, albeit with a need for careful hyperparameter tuning when state exploration interacts with reward design.

Abstract

Dynamic resource allocation in mobile wireless networks involves complex, time-varying optimization problems, motivating the adoption of deep reinforcement learning (DRL). However, most existing works rely on pre-trained policies, overlooking dynamic environmental changes that rapidly invalidate the policies. Periodic retraining becomes inevitable but incurs prohibitive computational costs and energy consumption-critical concerns for resource-constrained wireless systems. We identify three root causes of inefficient retraining: high-dimensional state spaces, suboptimal action spaces exploration-exploitation trade-offs, and reward design limitations. To overcome these limitations, we propose Diffusion-based Deep Reinforcement Learning (D2RL), which leverages generative diffusion models (GDMs) to holistically enhance all three DRL components. Iterative refinement process and distribution modelling of GDMs enable (1) the generation of diverse state samples to improve environmental understanding, (2) balanced action space exploration to escape local optima, and (3) the design of discriminative reward functions that better evaluate action quality. Our framework operates in two modes: Mode I leverages GDMs to explore reward spaces and design discriminative reward functions that rigorously evaluate action quality, while Mode II synthesizes diverse state samples to enhance environmental understanding and generalization. Extensive experiments demonstrate that D2RL achieves faster convergence and reduced computational costs over conventional DRL methods for resource allocation in wireless communications while maintaining competitive policy performance. This work underscores the transformative potential of GDMs in overcoming fundamental DRL training bottlenecks for wireless networks, paving the way for practical, real-time deployments.

Figures (9)

• Figure 1: Illustration of D2RL Framework.
• Figure 2: Comparison of (a) the sum rate achieved by GDM-based action exploration and without action exploration, and (b) MA reward obtained by GDM-based action exploration and without action exploration.
• Figure 3: Comparison of (a) total GPU time for convergence achieved by GDM-based action exploration and without action exploration, and (b) GPU time per epoch for different conditions achieved by GDM-based action exploration and without action exploration.
• Figure 4: Comparison of (a) the sum rate and (b) MA reward for different reward design cases.
• Figure 5: Comparison of (a) the mean gradient weight and (b) the mean gradient bias for different reward design cases.