Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

Soumyajit Guin, Shalabh Bhatnagar

TL;DR

The paper tackles stochastic shortest path problems by introducing two convergent tabular two-timescale algorithms (AC and CA) and one function-approximation-based AC algorithm (AC-FA). It provides rigorous convergence guarantees for all variants via two-timescale stochastic approximation, ODE/DI analyses, and Lyapunov arguments. Empirical results show the tabular methods outperform common convergent baselines (Q-Learning, SARSA), while the FA method demonstrates reliable convergence where alternatives may diverge or chatter. Overall, the work delivers principled, model-free SSP solutions with convergence guarantees and practical performance, and points to extensions like constrained optimization and Natural Actor-Critic approaches for broader applicability.

Abstract

In this paper we propose two algorithms in the tabular setting and an algorithm for the function approximation setting for the Stochastic Shortest Path (SSP) problem. SSP problems form an important class of problems in Reinforcement Learning (RL), as other types of cost-criteria in RL can be formulated in the setting of SSP. We show asymptotic almost-sure convergence for all our algorithms. We observe superior performance of our tabular algorithms compared to other well-known convergent RL algorithms. We further observe reliable performance of our function approximation algorithm compared to other algorithms in the function approximation setting.

Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

TL;DR

The paper tackles stochastic shortest path problems by introducing two convergent tabular two-timescale algorithms (AC and CA) and one function-approximation-based AC algorithm (AC-FA). It provides rigorous convergence guarantees for all variants via two-timescale stochastic approximation, ODE/DI analyses, and Lyapunov arguments. Empirical results show the tabular methods outperform common convergent baselines (Q-Learning, SARSA), while the FA method demonstrates reliable convergence where alternatives may diverge or chatter. Overall, the work delivers principled, model-free SSP solutions with convergence guarantees and practical performance, and points to extensions like constrained optimization and Natural Actor-Critic approaches for broader applicability.

Abstract

In this paper we propose two algorithms in the tabular setting and an algorithm for the function approximation setting for the Stochastic Shortest Path (SSP) problem. SSP problems form an important class of problems in Reinforcement Learning (RL), as other types of cost-criteria in RL can be formulated in the setting of SSP. We show asymptotic almost-sure convergence for all our algorithms. We observe superior performance of our tabular algorithms compared to other well-known convergent RL algorithms. We further observe reliable performance of our function approximation algorithm compared to other algorithms in the function approximation setting.

Paper Structure

This paper contains 13 sections, 19 theorems, 101 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contributions
  4. Preliminaries
  5. Actor-Critic and Critic-Actor Algorithms in the Tabular Setting
  6. Convergence Analysis of Algorithms in Tabular Setting
  7. Convergence of Actor Critic
  8. Convergence of Critic Actor
  9. Main Result
  10. An Actor-Critic Algorithm with Function Approximation
  11. Convergence Analysis of Algorithm in Function Approximation Setting
  12. Numerical Experiments
  13. Conclusions

Key Result

Lemma 1

$\sup_n ||V_n||<\infty$ a.s

Figures (5)

  • Figure 1: Random MDP with State space size $|S|=20$, Action space size $|A|=4$
  • Figure 2: FrozenLake gym Environemtnt with size 4x4(left) and 8x8(right)
  • Figure 3: Plots for comparison between AC-FA, Q-LFA and SARSA-LFA
  • Figure 4: MDP used for Q-Learning with LFA
  • Figure 5: MDP used for SARSA with LFA

Theorems & Definitions (45)

  • Remark 1
  • Remark 2
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Remark 3
  • Lemma 4
  • ...and 35 more