Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion Model-based Reinforcement Learning for Version Age of Information Scheduling: Average and Tail-Risk-Sensitive Control

Haoyuan Pan, Sizhao Chen, Zhaorui Wang, Tse-Tin Chan

TL;DR

This work tackles semantic staleness in multi-user VAoI scheduling under stochastic arrivals and unreliable channels by formulating a CMDP and proposing two DRL frameworks. The risk-neutral baseline, D2SAC, uses a diffusion-based actor to enrich policy generation and improve mean VAoI under a long-term transmission-cost constraint. The risk-sensitive extension, RS-D3SAC, couples a diffusion-based actor with a distributional QR-DQN critic to model the full VAoI return distribution and optimize CVaR, achieving substantial tail-risk reductions without sacrificing mean performance. Experimental results show that diffusion enhances mean performance, while the distributional critic drives the dominant gains in tail-risk control, yielding a robust, risk-aware VAoI scheduling approach for wireless networks.

Abstract

Ensuring timely and semantically accurate information delivery is critical in real-time wireless systems. While Age of Information (AoI) quantifies temporal freshness, Version Age of Information (VAoI) captures semantic staleness by accounting for version evolution between transmitters and receivers. Existing VAoI scheduling approaches primarily focus on minimizing average VAoI, overlooking rare but severe staleness events that can compromise reliability under stochastic packet arrivals and unreliable channels. This paper investigates both average-oriented and tail-risk-sensitive VAoI scheduling in a multi-user status update system with long-term transmission cost constraints. We first formulate the average VAoI minimization problem as a constrained Markov decision process and introduce a deep diffusion-based Soft Actor-Critic (D2SAC) algorithm. By generating actions through a diffusion-based denoising process, D2SAC enhances policy expressiveness and establishes a strong baseline for mean performance. Building on this foundation, we put forth RS-D3SAC, a risk-sensitive deep distributional diffusion-based Soft Actor-Critic algorithm. RS-D3SAC integrates a diffusion-based actor with a quantile-based distributional critic, explicitly modeling the full VAoI return distribution. This enables principled tail-risk optimization via Conditional Value-at-Risk (CVaR) while satisfying long-term transmission cost constraints. Extensive simulations show that, while D2SAC reduces average VAoI, RS-D3SAC consistently achieves substantial reductions in CVaR without sacrificing mean performance. The dominant gain in tail-risk reduction stems from the distributional critic, with the diffusion-based actor providing complementary refinement to stabilize and enrich policy decisions, highlighting their effectiveness for robust and risk-aware VAoI scheduling in multi-user wireless systems.

Diffusion Model-based Reinforcement Learning for Version Age of Information Scheduling: Average and Tail-Risk-Sensitive Control

TL;DR

This work tackles semantic staleness in multi-user VAoI scheduling under stochastic arrivals and unreliable channels by formulating a CMDP and proposing two DRL frameworks. The risk-neutral baseline, D2SAC, uses a diffusion-based actor to enrich policy generation and improve mean VAoI under a long-term transmission-cost constraint. The risk-sensitive extension, RS-D3SAC, couples a diffusion-based actor with a distributional QR-DQN critic to model the full VAoI return distribution and optimize CVaR, achieving substantial tail-risk reductions without sacrificing mean performance. Experimental results show that diffusion enhances mean performance, while the distributional critic drives the dominant gains in tail-risk control, yielding a robust, risk-aware VAoI scheduling approach for wireless networks.

Abstract

Ensuring timely and semantically accurate information delivery is critical in real-time wireless systems. While Age of Information (AoI) quantifies temporal freshness, Version Age of Information (VAoI) captures semantic staleness by accounting for version evolution between transmitters and receivers. Existing VAoI scheduling approaches primarily focus on minimizing average VAoI, overlooking rare but severe staleness events that can compromise reliability under stochastic packet arrivals and unreliable channels. This paper investigates both average-oriented and tail-risk-sensitive VAoI scheduling in a multi-user status update system with long-term transmission cost constraints. We first formulate the average VAoI minimization problem as a constrained Markov decision process and introduce a deep diffusion-based Soft Actor-Critic (D2SAC) algorithm. By generating actions through a diffusion-based denoising process, D2SAC enhances policy expressiveness and establishes a strong baseline for mean performance. Building on this foundation, we put forth RS-D3SAC, a risk-sensitive deep distributional diffusion-based Soft Actor-Critic algorithm. RS-D3SAC integrates a diffusion-based actor with a quantile-based distributional critic, explicitly modeling the full VAoI return distribution. This enables principled tail-risk optimization via Conditional Value-at-Risk (CVaR) while satisfying long-term transmission cost constraints. Extensive simulations show that, while D2SAC reduces average VAoI, RS-D3SAC consistently achieves substantial reductions in CVaR without sacrificing mean performance. The dominant gain in tail-risk reduction stems from the distributional critic, with the diffusion-based actor providing complementary refinement to stabilize and enrich policy decisions, highlighting their effectiveness for robust and risk-aware VAoI scheduling in multi-user wireless systems.
Paper Structure (23 sections, 43 equations, 11 figures, 2 tables, 2 algorithms)

This paper contains 23 sections, 43 equations, 11 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Freshness Metrics and Scheduling Approaches
  4. Diffusion Model-Based Reinforcement Learning
  5. Distributional Reinforcement Learning and Risk-Sensitive Wireless Optimization
  6. System Model and Performance Metrics
  7. System Model
  8. Performance Metrics
  9. Risk-Neutral Scheduling with Deep Diffusion Model-Based SAC
  10. Risk-Neutral Scheduling Problem Formulation
  11. Deep Diffusion Model-Based SAC
  12. Policy Generation by the Diffusion Model
  13. Integrating Diffusion Model Into the SAC Framework
  14. Risk-Sensitive Scheduling with Deep Distributional Diffusion Model-Based SAC
  15. Risk-Sensitive Scheduling Problem Formulation
  16. ...and 8 more sections

Figures (11)

  • Figure 1: System model: the scheduler receives packets from $N$ independent users and forwards them to the destination over unreliable wireless channels, selecting at most one user per time slot.
  • Figure 2: Example of a single user's VAoI evolution. Black arrows indicate packet generation, red arrows indicate successful packet transmission, and gray dashed arrows indicate failed packet transmission.
  • Figure 3: Schematic overview of the diffusion-based policy network.
  • Figure 4: Architecture of the D2SAC algorithm.
  • Figure 5: Architecture of the proposed RS-D3SAC algorithm.
  • ...and 6 more figures