Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Post-Decision State-Based Online Learning for Delay-Energy-Aware Flow Allocation in Wireless Systems

Mahesh Ganesh Bhat, Shana Moothedath, Prasanna Chaporkar

TL;DR

This work tackles delay- and energy-aware flow allocation across multiple 5G UPFs under stochastic and unknown traffic. It casts the problem as a Markov Decision Process and introduces a structure-aware, model-free reinforcement learning method based on post-decision states (PDS) to exploit the separation between controllable and exogenous dynamics. The proposed PDS-VI algorithm comes with convergence guarantees and demonstrates significantly faster learning and lower training cost than standard Q-learning, while achieving superior flow admission and lower long-term cost. This approach enables real-time, adaptive UPF flow allocation in dynamic 5G networks with uncertain arrivals and departures, improving both efficiency and QoS guarantees in practice.

Abstract

We develop a structure-aware reinforcement learning (RL) approach for delay- and energy-aware flow allocation in 5G User Plane Functions (UPFs). We consider a dynamic system with $K$ heterogeneous UPFs of varying capacities that handle stochastic arrivals of $M$ flow types, each with distinct rate requirements. We model the system as a Markov decision process (MDP) to capture the stochastic nature of flow arrivals and departures (possibly unknown), as well as the impact of flow allocation in the system. To solve this problem, we propose a post-decision state (PDS) based value iteration algorithm that exploits the underlying structure of the MDP. By separating action-controlled dynamics from exogenous factors, PDS enables faster convergence and efficient adaptive flow allocation, even in the absence of statistical knowledge about exogenous variables. Simulation results demonstrate that the proposed method converges faster and achieves lower long-term cost than standard Q-learning, highlighting the effectiveness of PDS-based RL for resource allocation in wireless networks.

Post-Decision State-Based Online Learning for Delay-Energy-Aware Flow Allocation in Wireless Systems

TL;DR

This work tackles delay- and energy-aware flow allocation across multiple 5G UPFs under stochastic and unknown traffic. It casts the problem as a Markov Decision Process and introduces a structure-aware, model-free reinforcement learning method based on post-decision states (PDS) to exploit the separation between controllable and exogenous dynamics. The proposed PDS-VI algorithm comes with convergence guarantees and demonstrates significantly faster learning and lower training cost than standard Q-learning, while achieving superior flow admission and lower long-term cost. This approach enables real-time, adaptive UPF flow allocation in dynamic 5G networks with uncertain arrivals and departures, improving both efficiency and QoS guarantees in practice.

Abstract

We develop a structure-aware reinforcement learning (RL) approach for delay- and energy-aware flow allocation in 5G User Plane Functions (UPFs). We consider a dynamic system with heterogeneous UPFs of varying capacities that handle stochastic arrivals of flow types, each with distinct rate requirements. We model the system as a Markov decision process (MDP) to capture the stochastic nature of flow arrivals and departures (possibly unknown), as well as the impact of flow allocation in the system. To solve this problem, we propose a post-decision state (PDS) based value iteration algorithm that exploits the underlying structure of the MDP. By separating action-controlled dynamics from exogenous factors, PDS enables faster convergence and efficient adaptive flow allocation, even in the absence of statistical knowledge about exogenous variables. Simulation results demonstrate that the proposed method converges faster and achieves lower long-term cost than standard Q-learning, highlighting the effectiveness of PDS-based RL for resource allocation in wireless networks.
Paper Structure (15 sections, 1 theorem, 30 equations, 2 figures, 1 algorithm)

This paper contains 15 sections, 1 theorem, 30 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries: Markov Decision Process
  3. Problem Formulation: Delay-Sensitive Flow Allocation in 5G User Plane Functions
  4. System Model
  5. Markov Decision Process Formulation
  6. State Space
  7. Action Space and Control Policy
  8. Cost Function
  9. System Dynamics
  10. Optimal Policy
  11. Proposed Structure Leveraging RL Approach
  12. Post-Decision State Based Learning
  13. Proposed Algorithm: PDS-Based Value Iteration
  14. Numerical Simulations
  15. Conclusion

Key Result

Theorem 1

The PDS value function iterates in Eq. eq:ivi converge to the optimal PDS value function, $\widetilde{V}_{t} \rightarrow \widetilde{V}^\star$.

Figures (2)

  • Figure 1: Illustration of PDS-based state transition.
  • Figure 2: PDS-based value iteration vs. Q-learning. Figure \ref{['fig:4upf_rbe_M']} presents plots for average relative Bellman error, RBE, Figure \ref{['fig:4upf_total_cost_M']} presents average cost, and Figure \ref{['fig:4upf_flow_drop_M']} represents the number of flows blocked with respect to iteration for 5 UPF case.

Theorems & Definitions (7)

  • Remark 1
  • Remark 2
  • Theorem 1
  • proof
  • Remark 3
  • Remark 4
  • Remark 5