Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semi-Gradient SARSA Routing with Theoretical Guarantee on Traffic Stability and Weight Convergence

Yidan Wu, Yu Yu, Jianan Zhang, Li Jin

TL;DR

This work tackles dynamic routing over parallel servers with unbounded state spaces by introducing a semi-gradient SARSA algorithm using linear function approximation for Q-value estimation. A Lyapunov-based drift analysis combined with an ODE-based stochastic approximation framework yields a joint convergence guarantee: the traffic state remains stable and the weight vector converges almost surely to the approximate optimum $w^*$, if and only if the system is stabilizable with $\lambda<\sum_n\mu_n$. The approach uses a softmax policy over a linear-approximate $Q$-function $\hat{Q}(x,a;w)$ built from basis functions, enabling interpretable routing decisions and avoiding brittle neural networks. Simulation on a TCP-like congestion control problem shows SGS converges much faster than neural-network SARSA and achieves a substantial reduction in the average cost compared with a join-the-shortest-queue baseline, with a small optimality gap. These results provide a provably stable, scalable RL framework for dynamic routing in networks and related parallel-server systems.

Abstract

We consider the traffic control problem of dynamic routing over parallel servers, which arises in a variety of engineering systems such as transportation and data transmission. We propose a semi-gradient, on-policy algorithm that learns an approximate optimal routing policy. The algorithm uses generic basis functions with flexible weights to approximate the value function across the unbounded state space. Consequently, the training process lacks Lipschitz continuity of the gradient, boundedness of the temporal-difference error, and a prior guarantee on ergodicity, which are the standard prerequisites in existing literature on reinforcement learning theory. To address this, we combine a Lyapunov approach and an ordinary differential equation-based method to jointly characterize the behavior of traffic state and approximation weights. Our theoretical analysis proves that the training scheme guarantees traffic state stability and ensures almost surely convergence of the weights to the approximate optimum. We also demonstrate via simulations that our algorithm attains significantly faster convergence than neural network-based methods with an insignificant approximation error.

Semi-Gradient SARSA Routing with Theoretical Guarantee on Traffic Stability and Weight Convergence

TL;DR

This work tackles dynamic routing over parallel servers with unbounded state spaces by introducing a semi-gradient SARSA algorithm using linear function approximation for Q-value estimation. A Lyapunov-based drift analysis combined with an ODE-based stochastic approximation framework yields a joint convergence guarantee: the traffic state remains stable and the weight vector converges almost surely to the approximate optimum , if and only if the system is stabilizable with . The approach uses a softmax policy over a linear-approximate -function built from basis functions, enabling interpretable routing decisions and avoiding brittle neural networks. Simulation on a TCP-like congestion control problem shows SGS converges much faster than neural-network SARSA and achieves a substantial reduction in the average cost compared with a join-the-shortest-queue baseline, with a small optimality gap. These results provide a provably stable, scalable RL framework for dynamic routing in networks and related parallel-server systems.

Abstract

We consider the traffic control problem of dynamic routing over parallel servers, which arises in a variety of engineering systems such as transportation and data transmission. We propose a semi-gradient, on-policy algorithm that learns an approximate optimal routing policy. The algorithm uses generic basis functions with flexible weights to approximate the value function across the unbounded state space. Consequently, the training process lacks Lipschitz continuity of the gradient, boundedness of the temporal-difference error, and a prior guarantee on ergodicity, which are the standard prerequisites in existing literature on reinforcement learning theory. To address this, we combine a Lyapunov approach and an ordinary differential equation-based method to jointly characterize the behavior of traffic state and approximation weights. Our theoretical analysis proves that the training scheme guarantees traffic state stability and ensures almost surely convergence of the weights to the approximate optimum. We also demonstrate via simulations that our algorithm attains significantly faster convergence than neural network-based methods with an insignificant approximation error.

Paper Structure

This paper contains 14 sections, 4 theorems, 58 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Motivation
  3. Related Work
  4. Methodology and Contribution
  5. Modeling and Formulation
  6. System modeling
  7. MDP formulation
  8. Learning algorithm
  9. Joint convergence guarantee
  10. Stochastic Boundedness and Ergodicity
  11. Stochastic Regularity Policy and Basis Function
  12. Proof of Theorem \ref{['thm: joint convergence']}
  13. Simulation-based validation
  14. Conclusions

Key Result

Theorem 1

(Joint convergence) Consider a parallel service system with arrival rate $\lambda>0$ and service rates $\mu_1,\mu_2,\ldots,\mu_N>0$ which satisfy $\lambda<\sum_{n=1}^N\mu_n$. Suppose that Assumptions assum: basis func and assum:eq_sumk=0 are satisfied. Then, the semi-gradient SARSA algorithm ensures

Figures (3)

  • Figure 1: A parallel service system.
  • Figure 2: A parallel service system.
  • Figure 3: The convergence performance comparison between SGS and NNS.

Theorems & Definitions (4)

  • Theorem 1
  • Lemma 1
  • Lemma 2
  • Lemma 3