Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Non-Asymptotic Guarantees for Average-Reward Q-Learning with Adaptive Stepsizes

Zaiwei Chen

TL;DR

This work provides the first non-asymptotic, last-iterate convergence analysis for average-reward Q-learning with asynchronous updates, using adaptive stepsizes as local clocks for each state-action pair. By reformulating the non-Markovian SA dynamics as a time-inhomogeneous Markovian SA and developing a Lyapunov Moreau-envelope framework, the authors prove $\tilde{O}(1/k)$ mean-square convergence to the optimal relative Q-function in the span seminorm $p_ ext{span}$, and, after centering, pointwise convergence to the centered optimal relative Q-function $\tilde{Q}^*$. They also establish that universal (non-adaptive) stepsizes can fail to converge to the correct target, while adaptive stepsizes act as implicit importance sampling to counteract asynchronous updates. The results rely on a time-varying almost-sure bound, conditioning arguments, and Markov chain concentration to control strong correlations, offering a methodology potentially applicable to a broad class of stochastic approximation algorithms with adaptive stepsizes. Overall, the paper advances the theoretical understanding of finite-time, last-iterate guarantees for classical RL algorithms in the challenging average-reward, asynchronous setting, with implications for practical, model-free learning. $$Q^*, r^*, \tilde{Q}^*, p_ ext{span}, \mathcal{H}(\cdot), \bar{\mathcal{H}}(\cdot)$$ are central quantities throughout the analysis, and the $\tilde{O}(1/k)$ rates highlight the efficiency of the proposed methods in large-sample regimes.

Abstract

This work presents the first finite-time analysis for the last-iterate convergence of average-reward Q-learning with an asynchronous implementation. A key feature of the algorithm we study is the use of adaptive stepsizes, which serve as local clocks for each state-action pair. We show that the iterates generated by this Q-learning algorithm converge at a rate of $O(1/k)$ (in the mean-square sense) to the optimal relative Q-function in the span seminorm. Moreover, by adding a centering step to the algorithm, we further establish pointwise mean-square convergence to a centered optimal relative Q-function, also at a rate of $O(1/k)$. To prove these results, we show that adaptive stepsizes are necessary, as without them, the algorithm fails to converge to the correct target. In addition, adaptive stepsizes can be interpreted as a form of implicit importance sampling that counteracts the effects of asynchronous updates. Technically, the use of adaptive stepsizes makes each Q-learning update depend on the entire sample history, introducing strong correlations and making the algorithm a non-Markovian stochastic approximation (SA) scheme. Our approach to overcoming this challenge involves (1) a time-inhomogeneous Markovian reformulation of non-Markovian SA, and (2) a combination of almost-sure time-varying bounds, conditioning arguments, and Markov chain concentration inequalities to break the strong correlations between the adaptive stepsizes and the iterates. The tools developed in this work are likely to be broadly applicable to the analysis of general SA algorithms with adaptive stepsizes.

Non-Asymptotic Guarantees for Average-Reward Q-Learning with Adaptive Stepsizes

TL;DR

This work provides the first non-asymptotic, last-iterate convergence analysis for average-reward Q-learning with asynchronous updates, using adaptive stepsizes as local clocks for each state-action pair. By reformulating the non-Markovian SA dynamics as a time-inhomogeneous Markovian SA and developing a Lyapunov Moreau-envelope framework, the authors prove mean-square convergence to the optimal relative Q-function in the span seminorm , and, after centering, pointwise convergence to the centered optimal relative Q-function . They also establish that universal (non-adaptive) stepsizes can fail to converge to the correct target, while adaptive stepsizes act as implicit importance sampling to counteract asynchronous updates. The results rely on a time-varying almost-sure bound, conditioning arguments, and Markov chain concentration to control strong correlations, offering a methodology potentially applicable to a broad class of stochastic approximation algorithms with adaptive stepsizes. Overall, the paper advances the theoretical understanding of finite-time, last-iterate guarantees for classical RL algorithms in the challenging average-reward, asynchronous setting, with implications for practical, model-free learning. are central quantities throughout the analysis, and the rates highlight the efficiency of the proposed methods in large-sample regimes.

Abstract

This work presents the first finite-time analysis for the last-iterate convergence of average-reward Q-learning with an asynchronous implementation. A key feature of the algorithm we study is the use of adaptive stepsizes, which serve as local clocks for each state-action pair. We show that the iterates generated by this Q-learning algorithm converge at a rate of (in the mean-square sense) to the optimal relative Q-function in the span seminorm. Moreover, by adding a centering step to the algorithm, we further establish pointwise mean-square convergence to a centered optimal relative Q-function, also at a rate of . To prove these results, we show that adaptive stepsizes are necessary, as without them, the algorithm fails to converge to the correct target. In addition, adaptive stepsizes can be interpreted as a form of implicit importance sampling that counteracts the effects of asynchronous updates. Technically, the use of adaptive stepsizes makes each Q-learning update depend on the entire sample history, introducing strong correlations and making the algorithm a non-Markovian stochastic approximation (SA) scheme. Our approach to overcoming this challenge involves (1) a time-inhomogeneous Markovian reformulation of non-Markovian SA, and (2) a combination of almost-sure time-varying bounds, conditioning arguments, and Markov chain concentration inequalities to break the strong correlations between the adaptive stepsizes and the iterates. The tools developed in this work are likely to be broadly applicable to the analysis of general SA algorithms with adaptive stepsizes.

Paper Structure

This paper contains 54 sections, 22 theorems, 234 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Related Literature
  4. Average-Reward Q-Learning.
  5. Discounted Q-Learning.
  6. Stochastic Approximation.
  7. From Average-Reward RL to the Seminorm Bellman Equation
  8. Background on Average-Reward RL
  9. The Seminorm Bellman Equation
  10. Q-Learning with Set Convergence
  11. Algorithm
  12. Finite-Time Analysis
  13. Q-Learning with Pointwise Convergence
  14. The Necessity of Using Adaptive Stepsizes
  15. Q-Learning with Universal Stepsizes: Provable Convergence to the Wrong Target
  16. ...and 39 more sections

Key Result

Lemma 2.2

For any $x \in \mathbb{R}^d$, we have $\mathop{\mathrm{arg\,min}}\limits_{c \in \mathbb{R}} \|x - c e\|_\infty=(\max_i x_i + \min_j x_j)/2$. As a result, the span seminorm of $x$ can be equivalently written as $p_\text{span}(x) = \min_{c \in \mathbb{R}} \|x - c e\|_\infty$.

Figures (4)

  • Figure 1: Convergence to $Q^*$ in $p_\text{span}(\cdot)$
  • Figure 2: Convergence to $\bar{Q}^*$ in $p_\text{span}(\cdot)$
  • Figure 3: Performance of the Output Policies
  • Figure 4: Discounted $Q$-Learning

Theorems & Definitions (26)

  • Definition 2.1
  • Lemma 2.2
  • Lemma 2.4
  • Theorem 3.2
  • Corollary 3.3
  • Example 4.1
  • Lemma 4.2
  • Theorem 4.3
  • Lemma 5.1
  • Proposition 5.2
  • ...and 16 more