Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces

Brahma S. Pavse, Matthew Zurek, Yudong Chen, Qiaomin Xie, Josiah P. Hanna

TL;DR

This work addresses the challenge of learning stable policies in online reinforcement learning over unbounded state spaces, where naive optimization of the average-cost objective yields unstable behavior. It introduces STOP, which combines Lyapunov-based cost shaping with state transformations to steer learning toward stability without sacrificing optimality. The authors prove that the shaping preserves the set of optimal policies under mild conditions and empirically demonstrate that STOP yields stable, competitive performance on queueing and traffic-control domains, often outperforming baselines that rely on transition dynamics. The approach offers practical guidance for incorporating domain knowledge through Lyapunov functions and coordinate-wise state compression to improve generalization in challenging, real-world-inspired RL problems.

Abstract

In many reinforcement learning (RL) applications, we want policies that reach desired states and then keep the controlled system within an acceptable region around the desired states over an indefinite period of time. This latter objective is called stability and is especially important when the state space is unbounded, such that the states can be arbitrarily far from each other and the agent can drift far away from the desired states. For example, in stochastic queuing networks, where queues of waiting jobs can grow without bound, the desired state is all-zero queue lengths. Here, a stable policy ensures queue lengths are finite while an optimal policy minimizes queue lengths. Since an optimal policy is also stable, one would expect that RL algorithms would implicitly give us stable policies. However, in this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies, i.e., policies that drift far away from the desired state. We attribute this instability to poor credit-assignment for destabilizing actions. We then introduce an approach based on two ideas: 1) a Lyapunov-based cost-shaping technique and 2) state transformations to the unbounded state space. We conduct an empirical study on various queueing networks and traffic signal control problems and find that our approach performs competitively against strong baselines with knowledge of the transition dynamics. Our code is available here: https://github.com/Badger-RL/STOP.

Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces

TL;DR

This work addresses the challenge of learning stable policies in online reinforcement learning over unbounded state spaces, where naive optimization of the average-cost objective yields unstable behavior. It introduces STOP, which combines Lyapunov-based cost shaping with state transformations to steer learning toward stability without sacrificing optimality. The authors prove that the shaping preserves the set of optimal policies under mild conditions and empirically demonstrate that STOP yields stable, competitive performance on queueing and traffic-control domains, often outperforming baselines that rely on transition dynamics. The approach offers practical guidance for incorporating domain knowledge through Lyapunov functions and coordinate-wise state compression to improve generalization in challenging, real-world-inspired RL problems.

Abstract

In many reinforcement learning (RL) applications, we want policies that reach desired states and then keep the controlled system within an acceptable region around the desired states over an indefinite period of time. This latter objective is called stability and is especially important when the state space is unbounded, such that the states can be arbitrarily far from each other and the agent can drift far away from the desired states. For example, in stochastic queuing networks, where queues of waiting jobs can grow without bound, the desired state is all-zero queue lengths. Here, a stable policy ensures queue lengths are finite while an optimal policy minimizes queue lengths. Since an optimal policy is also stable, one would expect that RL algorithms would implicitly give us stable policies. However, in this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies, i.e., policies that drift far away from the desired state. We attribute this instability to poor credit-assignment for destabilizing actions. We then introduce an approach based on two ideas: 1) a Lyapunov-based cost-shaping technique and 2) state transformations to the unbounded state space. We conduct an empirical study on various queueing networks and traffic signal control problems and find that our approach performs competitively against strong baselines with knowledge of the transition dynamics. Our code is available here: https://github.com/Badger-RL/STOP.
Paper Structure (33 sections, 4 theorems, 8 equations, 10 figures, 1 table, 1 algorithm)

This paper contains 33 sections, 4 theorems, 8 equations, 10 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Why is Achieving Stochastic Stability Difficult for Deep RL?
  5. STOP: STability and OPtimality
  6. Lyapunov-based Cost Shaping
  7. Intuition for Encouraging Stability
  8. Choosing a Lyapunov Function
  9. Theoretical Properties
  10. State Transformations
  11. Empirical Study
  12. Setup
  13. Stability Encouragement
  14. Main Results
  15. Ablation Studies
  16. ...and 18 more sections

Key Result

Proposition 5.0

For any policy $\pi$, Therefore $J^{\textup{S}}(\pi) \geq J^{\textup{O}}(\pi)$, and $J^{\textup{S}}(\pi) = J^{\textup{O}}(\pi)$ if and only if $\mathbb{E}_\pi [\ell(s_{T+1})] = o(T)$.

Figures (10)

  • Figure 1: $2$-queue network setting where a central server must select which of the two queues to serve. $Q_i$, $\lambda_i$, and $p_i$ are the queue length, arrival rate, and service probability of queue $i$ at each time-step. The image is taken from liu_rlqueueing_2019.
  • Figure 2: State transformations applied to $1$D original untransformed points. Extreme points (red) appear closer to samples that agent was trained on (blue), thereby mitigating extreme generalization burdens. The green circle's radius is $5$ units and provides a sense of proximity of the points.
  • Figure 3: Stability verification. (a) fraction of unstable actions taken by the agent over the course of training of stop ($c(s') + \ell_2(s') - \ell_2(s)$) and ppo$(c(s'))$ agents; lower is better. (b) and (c) state-visitation distribution of stop and ppo agents respectively; higher $\Pr(\text{in sub-grid})$ is better. Note that the empty region of (c) shows the failure of the ppo agent to visit the specified bounded region near the target state. All quantities were computed over $10$ trials on the $2$-queue setting from Figure \ref{['fig:2queue_example']}.
  • Figure 4: True optimality criterion vs. interaction time-steps on three server-allocation queue networks (top row) and three $N$-network model environments (bottom row). Lower is better. Algorithms are ppo vs. stop-$p$, where $p$ denotes the power of the Lyapunov function. All stop variants use the symloge state transformation. We also report the performance of maxweight (mw). Recall that unlike mw, stop does not know the transition dynamics. The iqmagarwal2021deep is computed of the performance metrics over $20$ trials with $95\%$ confidence intervals. The vertical axis of (f) is log-scaled. To enhance visibility we zoom into the plot, which hides performance of o in some cases since it was unstable. We refer the reader to Appendix \ref{['sec:more_empirical']} for the zoomed-out plots.
  • Figure 5: Ablations. True optimality criterion vs. interaction time-steps on $2$ queue setup with faulty connections. Lower is better. The iqmagarwal2021deep is computed of the performance metrics over $20$ trials with $95\%$ confidence intervals. The vertical axis of (b) is log-scaled. id: identity; sig: symsigmoid, ss: symsqrt; sl: symloge. Refer to Appendix \ref{['sec:more_empirical']} for the zoomed-out plots.
  • ...and 5 more figures

Theorems & Definitions (7)

  • Definition 3.1: Stochastic Stability
  • Proposition 5.0
  • Proposition 5.0
  • Proposition 1.0
  • proof
  • Proposition 1.0
  • proof