Implicit Q-Learning and SARSA: Liberating Policy Control from Step-Size Calibration

TL;DR

This work tackles the long-standing problem of step-size sensitivity in Q-learning and SARSA when using function approximation. It introduces implicit variants that reformulate updates as fixed-point equations, yielding an explicit update with adaptive step-size that scales as $1/(1+\beta_t\|\boldsymbol{\phi}_t\|_2^2)$, augmented by a Sherman–Morrison–Woodbury construction. The authors provide non-asymptotic error bounds showing that implicit methods enjoy substantially greater step-size tolerance and achieve stable convergence even with large or varying $\beta_t$, alongside finite-time guarantees for both Q-learning and SARSA. Empirically, implicit Q-learning and implicit SARSA demonstrate robust performance across discrete and continuous state spaces, substantially reducing sensitivity to step-size choices and performing well where standard methods fail. These results offer practical stability advantages and point toward broader applicability of implicit updates in RL with function approximation.

Abstract

Q-learning and SARSA are foundational reinforcement learning algorithms whose practical success depends critically on step-size calibration. Step-sizes that are too large can cause numerical instability, while step-sizes that are too small can lead to slow progress. We propose implicit variants of Q-learning and SARSA that reformulate their iterative updates as fixed-point equations. This yields an adaptive step-size adjustment that scales inversely with feature norms, providing automatic regularization without manual tuning. Our non-asymptotic analyses demonstrate that implicit methods maintain stability over significantly broader step-size ranges. Under favorable conditions, it permits arbitrarily large step-sizes while achieving comparable convergence rates. Empirical validation across benchmark environments spanning discrete and continuous state spaces shows that implicit Q-learning and SARSA exhibit substantially reduced sensitivity to step-size selection, achieving stable performance with step-sizes that would cause standard methods to fail.

Figures (3)

• Figure 1: Sensitivity of Q-learning and SARSA to step-size selection on Cliff Walking and Taxi environments. Top: Average cumulative reward versus step-size $\beta$ shows both algorithms maintain stability below certain thresholds ($\beta \approx 1.0$ for Cliff Walking, $\beta \approx 0.8$ for Taxi) but degrade sharply beyond them. Bottom: Cumulative reward at fixed iterations (3,000 for Cliff Walking, 30,000 for Taxi) reveals that conservative step-sizes, while stable, yield substantially inferior performance. Shaded regions represent standard errors over 50 runs.
• Figure 2: Performance comparison on Cliff Walking and Taxi environments. Top: Average cumulative reward versus step-size $\beta$ shows that while standard Q-learning (blue) and SARSA (green) degrade sharply beyond thresholds ($\beta \approx 1.0$ for Cliff Walking, $\beta \approx 0.8$ for Taxi), implicit Q-learning (orange) and implicit SARSA (red) maintain stable performance across the entire range up to $\beta = 2.0$. Bottom: Cumulative reward at fixed iterations (3,000 for Cliff Walking, 30,000 for Taxi) demonstrates that implicit variants achieve strong performance at large step-sizes where standard methods fail. Shaded regions represent standard errors over 50 runs.
• Figure 3: Performance on continuous state environments with function approximation. Learning curves (log episode length vs episode number) for Q-learning and SARSA on Acrobot and Mountain Car with decreasing step-sizes $\beta_t = \beta_0/(t+1)^{2/3}$. Lower values indicate better performance. At a small initial step-size ($\beta_0=1$), all methods perform similarly. At a large initial step-size ($\beta_0=10$ for Acrobot, $\beta_0=5$ for Mountain Car), implicit methods (orange/red) demonstrate superior stability compared to standard methods (blue/green). Shaded regions show standard errors over 20 runs.

Theorems & Definitions (26)

• Proposition 3.1: Implicit Q-learning update
• Proposition 3.2: Implicit SARSA update
• Theorem 4.3: Finite-time bounds for Q-learning under constant step-size
• Theorem 4.4: Finite-time bounds for Q-learning under decreasing step-size
• Remark 4.5: Comparison on step-size condition
• Remark 4.6: Comparison on convergence rate and error terms
• Theorem 4.9: Finite-time bound for implicit SARSA under constant step-size
• Theorem 4.10: Finite-time bounds for SARSA under decreasing step-size
• Remark 4.11: Comparison on step-size conditions
• Remark 4.12: Comparison on convergence rate and error terms