Goal-oriented Transmission Scheduling: Structure-guided DRL with a Unified Dual On-policy and Off-policy Approach

TL;DR

This work targets goal-oriented transmission scheduling in multi-device, multi-channel wireless systems, where application-driven costs are governed by AoI and channel states. It derives key structural properties of the optimal policy and value function, including monotonicity w.r.t. channel states and asymptotic AoI convexity, and shows a greedy structure under certain co-located-device conditions. Building on these insights, the authors introduce SUDO-DRL, a structure-guided unified dual on-off policy that blends on-policy stability (PPO) with off-policy sample efficiency, augmented by a structural-property evaluation framework and selective replay buffering. Empirical results demonstrate substantial gains (up to ~45% performance improvement) and faster convergence, with robust scalability to large-scale systems (up to 40 devices and 20 channels) where traditional off-policy methods struggle. The approach advances goal-oriented communications by tightly integrating domain structure into DRL design, enabling scalable, efficient optimization of resource allocation under AoI-aware objectives.

Abstract

Goal-oriented communications prioritize application-driven objectives over data accuracy, enabling intelligent next-generation wireless systems. Efficient scheduling in multi-device, multi-channel systems poses significant challenges due to high-dimensional state and action spaces. We address these challenges by deriving key structural properties of the optimal solution to the goal-oriented scheduling problem, incorporating Age of Information (AoI) and channel states. Specifically, we establish the monotonicity of the optimal state value function (a measure of long-term system performance) w.r.t. channel states and prove its asymptotic convexity w.r.t. AoI states. Additionally, we derive the monotonicity of the optimal policy w.r.t. channel states, advancing the theoretical framework for optimal scheduling. Leveraging these insights, we propose the structure-guided unified dual on-off policy DRL (SUDO-DRL), a hybrid algorithm that combines the stability of on-policy training with the sample efficiency of off-policy methods. Through a novel structural property evaluation framework, SUDO-DRL enables effective and scalable training, addressing the complexities of large-scale systems. Numerical results show SUDO-DRL improves system performance by up to 45% and reduces convergence time by 40% compared to state-of-the-art methods. It also effectively handles scheduling in much larger systems, where off-policy DRL fails and on-policy benchmarks exhibit significant performance loss, demonstrating its scalability and efficacy in goal-oriented communications.

Figures (7)

• Figure 1: Goal-oriented communication system with $N$ edge devices, $M$ channels, and a remote destination
• Figure 2: Critic and Actor NNs’ structural property evaluation framework.
• Figure 3: SUDO-DRL Architecture.
• Figure 4: Average cost during training with $N\!\!=\!40, M\!\!=\!20$.
• Figure 5: Critic monotonicity (CM) score during training with $N\!\!=\!40, M\!\!=\!20$.

Theorems & Definitions (24)

• Example 1: Remote state estimation system
• Lemma 1: Monotonicity of optimal V function w.r.t. AoI states chen2022seDRL
• Theorem 1: Monotonicity of the optimal V function w.r.t. channel states
• proof
• Definition 1: Discrete convexity of optimal V function and cost function w.r.t. AoI
• Lemma 2: Asymptotic convexity of the cost function w.r.t. AoI in a remote state estimation system of Example \ref{['example: remote estimation']}
• proof
• Theorem 2: Convexity of the optimal V function w.r.t. AoI of a two-device-one-channel systems
• proof
• Theorem 3: Asymptotic convexity of the optimal V function w.r.t. AoI of a multi-device-multi-channel system