Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Hybrid Reinforcement Learning Framework for Hard Latency Constrained Resource Scheduling

Luyuan Zhang, An Liu, Kexuan Wang

TL;DR

This work tackles resource scheduling for XR-enabled URLLC under burst traffic and hard latency constraints in a 6G setting. It introduces HRL-RSHLC, a hybrid reinforcement learning framework that mixes old policies, a domain-knowledge policy, and a new policy to maximize hard-latency constrained effective throughput (HLC-ET) without relying on CMDP formulations. The authors prove convergence to KKT points and demonstrate, via simulations, that HRL-RSHLC achieves faster convergence and higher throughput than multiple baselines, including CMDP-based approaches. The framework reduces the action space by controlling priority weights and leverages policy reuse, experience replay, and SSCA-based optimization to handle sparse rewards, achieving robust performance even under imperfect CSI. This approach offers a practical, theoretically grounded path to fast-converging, robust resource scheduling for URLLC in complex bursty traffic scenarios.

Abstract

In the forthcoming 6G era, extend reality (XR) has been regarded as an emerging application for ultra-reliable and low latency communications (URLLC) with new traffic characteristics and more stringent requirements. In addition to the quasi-periodical traffic in XR, burst traffic with both large frame size and random arrivals in some real world low latency communication scenarios has become the leading cause of network congestion or even collapse, and there still lacks an efficient algorithm for the resource scheduling problem under burst traffic with hard latency constraints. We propose a novel hybrid reinforcement learning framework for resource scheduling with hard latency constraints (HRL-RSHLC), which reuses polices from both old policies learned under other similar environments and domain-knowledge-based (DK) policies constructed using expert knowledge to improve the performance. The joint optimization of the policy reuse probabilities and new policy is formulated as an Markov Decision Problem (MDP), which maximizes the hard-latency constrained effective throughput (HLC-ET) of users. We prove that the proposed HRL-RSHLC can converge to KKT points with an arbitrary initial point. Simulations show that HRL-RSHLC can achieve superior performance with faster convergence speed compared to baseline algorithms.

A Hybrid Reinforcement Learning Framework for Hard Latency Constrained Resource Scheduling

TL;DR

This work tackles resource scheduling for XR-enabled URLLC under burst traffic and hard latency constraints in a 6G setting. It introduces HRL-RSHLC, a hybrid reinforcement learning framework that mixes old policies, a domain-knowledge policy, and a new policy to maximize hard-latency constrained effective throughput (HLC-ET) without relying on CMDP formulations. The authors prove convergence to KKT points and demonstrate, via simulations, that HRL-RSHLC achieves faster convergence and higher throughput than multiple baselines, including CMDP-based approaches. The framework reduces the action space by controlling priority weights and leverages policy reuse, experience replay, and SSCA-based optimization to handle sparse rewards, achieving robust performance even under imperfect CSI. This approach offers a practical, theoretically grounded path to fast-converging, robust resource scheduling for URLLC in complex bursty traffic scenarios.

Abstract

In the forthcoming 6G era, extend reality (XR) has been regarded as an emerging application for ultra-reliable and low latency communications (URLLC) with new traffic characteristics and more stringent requirements. In addition to the quasi-periodical traffic in XR, burst traffic with both large frame size and random arrivals in some real world low latency communication scenarios has become the leading cause of network congestion or even collapse, and there still lacks an efficient algorithm for the resource scheduling problem under burst traffic with hard latency constraints. We propose a novel hybrid reinforcement learning framework for resource scheduling with hard latency constraints (HRL-RSHLC), which reuses polices from both old policies learned under other similar environments and domain-knowledge-based (DK) policies constructed using expert knowledge to improve the performance. The joint optimization of the policy reuse probabilities and new policy is formulated as an Markov Decision Problem (MDP), which maximizes the hard-latency constrained effective throughput (HLC-ET) of users. We prove that the proposed HRL-RSHLC can converge to KKT points with an arbitrary initial point. Simulations show that HRL-RSHLC can achieve superior performance with faster convergence speed compared to baseline algorithms.

Paper Structure

This paper contains 35 sections, 2 theorems, 30 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related Works
  4. Non-RL methods
  5. RL methods
  6. Contributions
  7. System Model
  8. Downlink Multi-user MIMO (MU-MIMO) Transmission
  9. Traffic and Queue Dynamic Model
  10. Hard-Latency Constrained Effective Throughput
  11. Hybrid RL Framework and Problem Formulation
  12. Resource Scheduling Based on Weighted Sum-Rate Maximization
  13. Problem Formulation based on MDP
  14. Hybrid RL Framework
  15. Preliminary on the Policy Reuse
  16. ...and 20 more sections

Key Result

Lemma 1

(Asymptotic consistency of surrogate functions:) Suppose that Assumptions 1 and 2 are satisfied, we have

Figures (8)

  • Figure 1: System model
  • Figure 2: Illustration of a strongly convex rate region.
  • Figure 3: Comparison with CMDP-based HRL-RSHLC.
  • Figure 4: Comparison with HRL-RSHLC that directly controls the user scheduling and power allocation.
  • Figure 5: Performance comparisons for different cases when $K=8$, $N_{t}=4$.
  • ...and 3 more figures

Theorems & Definitions (4)

  • Remark 1
  • Lemma 1
  • Theorem 1
  • Remark 2