Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving $\varepsilon^{-2}$ Dependence for Average-Reward Q-Learning with a New Contraction Principle

Zijun Chen, Zaiwei Chen, Nian Si, Shengbo Wang

TL;DR

This work addresses the lack of contraction in average-reward Q-learning by introducing a lazy transformation of the MDP and a problem-dependent seminorm \\widetilde{sp}(⋅). Under a reachability assumption, the authors prove that the lazy Bellman operator contracts one step in \\widetilde{sp}, enabling finite-sample, last-iterate convergence guarantees for both synchronous and asynchronous Q-learning with \\widetilde{O}(\\varepsilon^{-2}) sample complexity. The proposed methods remain model-free and simple, using explicit or implicit lazy sampling and suitable output correction to recover Q^* and derive \\varepsilon-optimal policies with high probability. The results combine a novel contraction principle with Lyapunov-based analysis, and are corroborated by numerical experiments showing the expected \\varepsilon^{-2} scaling and practical performance gains for the lazy variants. Overall, this work advances understanding of finite-time behavior in average-reward RL and offers practical, near-optimal sample complexity guarantees without assuming contraction of the original Bellman operator.

Abstract

We present the convergence rates of synchronous and asynchronous Q-learning for average-reward Markov decision processes, where the absence of contraction poses a fundamental challenge. Existing non-asymptotic results overcome this challenge by either imposing strong assumptions to enforce seminorm contraction or relying on discounted or episodic Markov decision processes as successive approximations, which either require unknown parameters or result in suboptimal sample complexity. In this work, under a reachability assumption, we establish optimal $\widetilde{O}(\varepsilon^{-2})$ sample complexity guarantees (up to logarithmic factors) for a simple variant of synchronous and asynchronous Q-learning that samples from the lazified dynamics, where the system remains in the current state with some fixed probability. At the core of our analysis is the construction of an instance-dependent seminorm and showing that, after a lazy transformation of the Markov decision process, the Bellman operator becomes one-step contractive under this seminorm.

Achieving $\varepsilon^{-2}$ Dependence for Average-Reward Q-Learning with a New Contraction Principle

TL;DR

This work addresses the lack of contraction in average-reward Q-learning by introducing a lazy transformation of the MDP and a problem-dependent seminorm \\widetilde{sp}(⋅). Under a reachability assumption, the authors prove that the lazy Bellman operator contracts one step in \\widetilde{sp}, enabling finite-sample, last-iterate convergence guarantees for both synchronous and asynchronous Q-learning with \\widetilde{O}(\\varepsilon^{-2}) sample complexity. The proposed methods remain model-free and simple, using explicit or implicit lazy sampling and suitable output correction to recover Q^* and derive \\varepsilon-optimal policies with high probability. The results combine a novel contraction principle with Lyapunov-based analysis, and are corroborated by numerical experiments showing the expected \\varepsilon^{-2} scaling and practical performance gains for the lazy variants. Overall, this work advances understanding of finite-time behavior in average-reward RL and offers practical, near-optimal sample complexity guarantees without assuming contraction of the original Bellman operator.

Abstract

We present the convergence rates of synchronous and asynchronous Q-learning for average-reward Markov decision processes, where the absence of contraction poses a fundamental challenge. Existing non-asymptotic results overcome this challenge by either imposing strong assumptions to enforce seminorm contraction or relying on discounted or episodic Markov decision processes as successive approximations, which either require unknown parameters or result in suboptimal sample complexity. In this work, under a reachability assumption, we establish optimal sample complexity guarantees (up to logarithmic factors) for a simple variant of synchronous and asynchronous Q-learning that samples from the lazified dynamics, where the system remains in the current state with some fixed probability. At the core of our analysis is the construction of an instance-dependent seminorm and showing that, after a lazy transformation of the Markov decision process, the Bellman operator becomes one-step contractive under this seminorm.
Paper Structure (48 sections, 42 theorems, 323 equations, 3 figures, 1 table, 4 algorithms)

This paper contains 48 sections, 42 theorems, 323 equations, 3 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Preliminaries
  4. Reachability of a reference state.
  5. Bellman operator and optimality equation.
  6. One-Step Contraction via Lazy Transformation
  7. Lazy Transformation
  8. Seminorm Contraction
  9. Synchronous Q-learning
  10. Asynchronous Q-learning
  11. Numerical Experiments
  12. Conclusion and Future Work
  13. Notation and Basic Properties
  14. Finiteness of the Hitting Time
  15. Technical Results in Section \ref{['sec:one_step_contraction_via_lazy_transformation']}
  16. ...and 33 more sections

Key Result

Lemma 3.1

Let $\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r)$ be an MDP. For any $\alpha \in (0,1]$, let $\overline{P}$ be defined as in eq:def_lazy_kernel. Let $(g^*, Q^*)$ be a solution to the average-reward Bellman equation equ:bellman_equation under $P$. Then, for all $(s,a)\in \mathcal{S}\times \mathcal solves the Bellman equation under $\overline P$, i.e. Moreover, the optimal policy set is preserve

Figures (3)

  • Figure 1: Convergence of lazy Q-learning algorithms with $p=0.3$ and $q=0.7$. The figures show the last-iterate span error against the sample complexity.
  • Figure 2: A hierarchy of MDP classes and assumptions.
  • Figure 3: Four-state MDP in diamond layout with two actions.

Theorems & Definitions (68)

  • Lemma 3.1: Lazy Transformation
  • Definition 3.2: Instance-Dependent Seminorm
  • Proposition 3.3
  • Theorem 3.4: Contraction in $\widetilde{sp}$
  • Remark 4.1
  • Theorem 4.2: Sample Complexity of Synchronous Q-learning
  • Theorem 5.1: Sample Complexity of Algorithm \ref{['alg:async_q_explicit_lazy_sampling']}
  • Theorem 5.2: Sample Complexity of Algorithm \ref{['alg:async_q_implicit_lazy_sampling']}
  • Remark 5.3
  • Lemma A.1
  • ...and 58 more