Offline Critic-Guided Diffusion Policy for Multi-User Delay-Constrained Scheduling
Zhuoran Li, Ruishuo Chen, Hai Zhong, Longbo Huang
TL;DR
The paper tackles multi-user delay-constrained scheduling under strict resource budgets when system dynamics are time-varying and unknown. It introduces SOCD, an offline reinforcement learning framework that combines a diffusion-based policy for rich, multi-modal action modeling with a sampling-free critic to guide policy improvement, all learned from pre-collected offline data. By embedding delayed-aware queues and a Lagrangian dual into an offline MDP, SOCD enforces both delay and resource constraints without system interaction during training, offering robustness to partial observability and large-scale networks. Empirical results across single-hop, multi-hop, and real-data scenarios show SOCD consistently outperforms offline baselines like SOLAR and BC, while maintaining stable resource usage and scalability, highlighting its practical impact for real-world delay-constrained scheduling.
Abstract
Effective multi-user delay-constrained scheduling is crucial in various real-world applications, such as instant messaging, live streaming, and data center management. In these scenarios, schedulers must make real-time decisions to satisfy both delay and resource constraints without prior knowledge of system dynamics, which are often time-varying and challenging to estimate. Current learning-based methods typically require interactions with actual systems during the training stage, which can be difficult or impractical, as it is capable of significantly degrading system performance and incurring substantial service costs. To address these challenges, we propose a novel offline reinforcement learning-based algorithm, named \underline{S}cheduling By \underline{O}ffline Learning with \underline{C}ritic Guidance and \underline{D}iffusion Generation (SOCD), to learn efficient scheduling policies purely from pre-collected \emph{offline data}. SOCD innovatively employs a diffusion-based policy network, complemented by a sampling-free critic network for policy guidance. By integrating the Lagrangian multiplier optimization into the offline reinforcement learning, SOCD effectively trains high-quality constraint-aware policies exclusively from available datasets, eliminating the need for online interactions with the system. Experimental results demonstrate that SOCD is resilient to various system dynamics, including partially observable and large-scale environments, and delivers superior performance compared to existing methods.