Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Minimal-Assumption Analysis of Q-Learning with Time-Varying Policies

Phalguni Nanda, Zaiwei Chen

TL;DR

The study provides the first finite-time analysis of Q-learning with time-varying (on-policy) sampling under a minimal irreducibility assumption, establishing a last-iterate convergence rate for $\mathbb{E}[\|Q_k - Q^*\|_\infty^2]$ and a corresponding sample complexity of $\tilde{O}(1/\epsilon^2)$. It introduces a Poisson-equation-based decomposition of time-inhomogeneous Markovian noise and uses a lazy-chain sensitivity analysis to derive explicit bounds, linking the learning dynamics to fundamental exploration properties of the MDP. A second result characterizes the convergence of the learning policy by bounding $\mathbb{E}[\|Q^{\pi_k}-Q^*\|_\infty^2]$, highlighting an exploitation advantage as $\pi_k$ converges to an optimal policy, while on-policy exploration is inherently weaker than off-policy. Numerical simulations corroborate the theory, showing slower convergence for on-policy Q-learning but task-appropriate policy convergence, and they illustrate the practical impact of the exploration–exploitation trade-off on on-policy learning. The techniques developed—Poisson-equation decomposition and lazy-chain sensitivity—offer a framework for analyzing RL methods with rapidly varying learning policies and may extend to single-timescale actor–critic and learning-in-games algorithms.

Abstract

In this work, we present the first finite-time analysis of the Q-learning algorithm under time-varying learning policies (i.e., on-policy sampling) with minimal assumptions -- specifically, assuming only the existence of a policy that induces an irreducible Markov chain over the state space. We establish a last-iterate convergence rate for $\mathbb{E}[\|Q_k - Q^*\|_\infty^2]$, implying a sample complexity of order $O(1/ε^2)$ for achieving $\mathbb{E}[\|Q_k - Q^*\|_\infty] \le ε$, matching that of off-policy Q-learning but with a worse dependence on exploration-related parameters. We also derive an explicit rate for $\mathbb{E}[\|Q^{π_k} - Q^*\|_\infty^2]$, where $π_k$ is the learning policy at iteration $k$. These results reveal that on-policy Q-learning exhibits weaker exploration than its off-policy counterpart but enjoys an exploitation advantage, as its policy converges to an optimal one rather than remaining fixed. Numerical simulations corroborate our theory. Technically, the combination of time-varying learning policies (which induce rapidly time-inhomogeneous Markovian noise) and the minimal assumption on exploration presents significant analytical challenges. To address these challenges, we employ a refined approach that leverages the Poisson equation to decompose the Markovian noise corresponding to the lazy transition matrix into a martingale-difference term and residual terms. To control the residual terms under time inhomogeneity, we perform a sensitivity analysis of the Poisson equation solution with respect to both the Q-function estimate and the learning policy. These tools may further facilitate the analysis of general reinforcement learning algorithms with rapidly time-varying learning policies -- such as single-timescale actor--critic methods and learning-in-games algorithms -- and are of independent interest.

A Minimal-Assumption Analysis of Q-Learning with Time-Varying Policies

TL;DR

The study provides the first finite-time analysis of Q-learning with time-varying (on-policy) sampling under a minimal irreducibility assumption, establishing a last-iterate convergence rate for and a corresponding sample complexity of . It introduces a Poisson-equation-based decomposition of time-inhomogeneous Markovian noise and uses a lazy-chain sensitivity analysis to derive explicit bounds, linking the learning dynamics to fundamental exploration properties of the MDP. A second result characterizes the convergence of the learning policy by bounding , highlighting an exploitation advantage as converges to an optimal policy, while on-policy exploration is inherently weaker than off-policy. Numerical simulations corroborate the theory, showing slower convergence for on-policy Q-learning but task-appropriate policy convergence, and they illustrate the practical impact of the exploration–exploitation trade-off on on-policy learning. The techniques developed—Poisson-equation decomposition and lazy-chain sensitivity—offer a framework for analyzing RL methods with rapidly varying learning policies and may extend to single-timescale actor–critic and learning-in-games algorithms.

Abstract

In this work, we present the first finite-time analysis of the Q-learning algorithm under time-varying learning policies (i.e., on-policy sampling) with minimal assumptions -- specifically, assuming only the existence of a policy that induces an irreducible Markov chain over the state space. We establish a last-iterate convergence rate for , implying a sample complexity of order for achieving , matching that of off-policy Q-learning but with a worse dependence on exploration-related parameters. We also derive an explicit rate for , where is the learning policy at iteration . These results reveal that on-policy Q-learning exhibits weaker exploration than its off-policy counterpart but enjoys an exploitation advantage, as its policy converges to an optimal one rather than remaining fixed. Numerical simulations corroborate our theory. Technically, the combination of time-varying learning policies (which induce rapidly time-inhomogeneous Markovian noise) and the minimal assumption on exploration presents significant analytical challenges. To address these challenges, we employ a refined approach that leverages the Poisson equation to decompose the Markovian noise corresponding to the lazy transition matrix into a martingale-difference term and residual terms. To control the residual terms under time inhomogeneity, we perform a sensitivity analysis of the Poisson equation solution with respect to both the Q-function estimate and the learning policy. These tools may further facilitate the analysis of general reinforcement learning algorithms with rapidly time-varying learning policies -- such as single-timescale actor--critic methods and learning-in-games algorithms -- and are of independent interest.
Paper Structure (40 sections, 20 theorems, 103 equations, 4 figures, 1 algorithm)

This paper contains 40 sections, 20 theorems, 103 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Related Literature
  4. Organization.
  5. Background
  6. Reinforcement Learning
  7. Q-Learning with Time-Varying Learning Policies
  8. Main Results
  9. Finite-Time Analysis
  10. Proof of Theorem \ref{['thm1']}
  11. Stochastic Approximation under Rapidly Time-Inhomogeneous Markovian noise
  12. A Lyapunov Drift Approach for Error Decomposition
  13. Handling the Time-Inhomogeneous Markovian noise: A Poisson Equation Approach
  14. The Poisson Equation
  15. Sensitivity Analysis Based on the Lazy Chain
  16. ...and 25 more sections

Key Result

Theorem 3.3

Suppose that Assumption as:MC is satisfied, the stepsize $\alpha< 1/c_1$, $\epsilon\in (0,1]$ and $\tau\in (0,1/(1-\gamma)]$. Then, the following inequality holds for all $k \geq 0$: where with $\lambda:=\min_{0\leq n\leq k}\min_{s,a}\pi_n(a|s)\geq \epsilon/|\mathcal{A}|$ and $c_2',c_3'$ being absolute constants.

Figures (4)

  • Figure 1: The MDP structure.
  • Figure 2: Convergence rates of $Q_k$.
  • Figure 3: Convergence rates of $Q^{\pi_k}$.
  • Figure 4: The exploration–exploitation trade-off.

Theorems & Definitions (22)

  • Remark 3.2
  • Theorem 3.3
  • Corollary 3.4
  • Theorem 3.5
  • Lemma 4.1
  • Lemma 4.2
  • Lemma 4.3
  • Lemma 4.4: Proposition 1 from chen2024lyapunov
  • Lemma 4.5
  • Lemma 4.6
  • ...and 12 more