Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimistic Training and Convergence of Q-Learning -- Extended Version

Prashant Mehta, Sean Meyn

TL;DR

The paper investigates the convergence behavior of Q-learning with linear function approximation under an $(\varepsilon,\kappa)$-tamed Gibbs policy, focusing on stability, existence of a projected Bellman equation (PBE) solution, and conditions for convergence beyond tabular/linear MDPs. Using an ODE-based mean-flow framework and stochastic approximation, it shows that ultimate boundedness of the parameter iterates can be achieved and a PBE solution can exist, but uniqueness and convergence are not guaranteed. Through one- and two-dimensional counterexamples and numerical experiments, the authors demonstrate that oblivious training policies can destabilize learning and that even in richer function classes multiple PBE (or MSBE) equilibria can arise, leading to potential convergence to different roots depending on initialization and policy. The findings highlight fundamental limitations in existing convergence claims for linear-Q-learning, underscoring the need for stronger basis design and policy-construction guidelines rather than relying on standard oblivious or simple exploration schemes. These results have practical implications for designing stable and reliable reinforcement learning algorithms in settings with function approximation.

Abstract

In recent work it is shown that Q-learning with linear function approximation is stable, in the sense of bounded parameter estimates, under the $(\varepsilon,κ)$-tamed Gibbs policy; $κ$ is inverse temperature, and $\varepsilon>0$ is introduced for additional exploration. Under these assumptions it also follows that there is a solution to the projected Bellman equation (PBE). Left open is uniqueness of the solution, and criteria for convergence outside of the standard tabular or linear MDP settings. The present work extends these results to other variants of Q-learning, and clarifies prior work: a one dimensional example shows that under an oblivious policy for training there may be no solution to the PBE, or multiple solutions, and in each case the algorithm is not stable under oblivious training. The main contribution is that far more structure is required for convergence. An example is presented for which the basis is ideal, in the sense that the true Q-function is in the span of the basis. However, there are two solutions to the PBE under the greedy policy, and hence also for the $(\varepsilon,κ)$-tamed Gibbs policy for all sufficiently small $\varepsilon>0$ and $κ\ge 1$.

Optimistic Training and Convergence of Q-Learning -- Extended Version

TL;DR

The paper investigates the convergence behavior of Q-learning with linear function approximation under an -tamed Gibbs policy, focusing on stability, existence of a projected Bellman equation (PBE) solution, and conditions for convergence beyond tabular/linear MDPs. Using an ODE-based mean-flow framework and stochastic approximation, it shows that ultimate boundedness of the parameter iterates can be achieved and a PBE solution can exist, but uniqueness and convergence are not guaranteed. Through one- and two-dimensional counterexamples and numerical experiments, the authors demonstrate that oblivious training policies can destabilize learning and that even in richer function classes multiple PBE (or MSBE) equilibria can arise, leading to potential convergence to different roots depending on initialization and policy. The findings highlight fundamental limitations in existing convergence claims for linear-Q-learning, underscoring the need for stronger basis design and policy-construction guidelines rather than relying on standard oblivious or simple exploration schemes. These results have practical implications for designing stable and reliable reinforcement learning algorithms in settings with function approximation.

Abstract

In recent work it is shown that Q-learning with linear function approximation is stable, in the sense of bounded parameter estimates, under the -tamed Gibbs policy; is inverse temperature, and is introduced for additional exploration. Under these assumptions it also follows that there is a solution to the projected Bellman equation (PBE). Left open is uniqueness of the solution, and criteria for convergence outside of the standard tabular or linear MDP settings. The present work extends these results to other variants of Q-learning, and clarifies prior work: a one dimensional example shows that under an oblivious policy for training there may be no solution to the PBE, or multiple solutions, and in each case the algorithm is not stable under oblivious training. The main contribution is that far more structure is required for convergence. An example is presented for which the basis is ideal, in the sense that the true Q-function is in the span of the basis. However, there are two solutions to the PBE under the greedy policy, and hence also for the -tamed Gibbs policy for all sufficiently small and .
Paper Structure (12 sections, 6 theorems, 42 equations, 6 figures)

This paper contains 12 sections, 6 theorems, 42 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Optimism and Stability
  3. Preliminaries
  4. Challenge with oblivious policies
  5. Stability with optimism
  6. Limits of Q learning
  7. Extensions to SGD
  8. Can We Expect Convergence?
  9. Q-learning in dimension two
  10. Numerical examples
  11. Q-learning for MSBE
  12. Conclusions

Key Result

Lemma 2.1

[lemma]t:Aoblivious Suppose that the input used for training is oblivious, and that $\{Z_k = (X_k,U_k) : k\ge 0 \}$ has a unique steady-state pmf $\upvarpi$ on the finite state-action space ${\sf X}\times{\sf U}$. Then, $\sqcap$$\sqcup$=0

Figures (6)

  • Figure 1: Geometry of the mean flow $\bar{f}$ and sample Q-learning trajectories under the tamed Gibbs policy ($\kappa=20$, $\varepsilon=10^{-3}$). (a) Norm $\|\bar{f}(\theta) \|$, revealing two basins associated with distinct PBE equilibria. (b) Mean–flow vectors and solution curves, together with the greedy–policy equilibria ${\theta^\star,\theta^\blacklozenge }$ and the tamed–Gibbs equilibrium $\theta^{\textsf{\footnotesize t}}$. (c) Ten Q-learning trajectories from a common initial condition: one approaches $\theta^{\textsf{\footnotesize t}}$, while the others converge to $\theta^\star$.
  • Figure 2: The mean-flow vector field in case 2 of \ref{['t:Qprobs_Oblivious']}.
  • Figure 3: The function $\bar{f}$ illustrating each of the three cases of \ref{['t:Qprobs_Oblivious']}, based on \ref{['e:linEx']}, with parameters (a)$m=-10$, $r=0.1$, $w = 1$. (b)$r=-m=1$, $w = 10$. (c)$r=-m=1$, $w = -10$. In each case the mean flow is EAS under the tamed Gibbs policy, but the oblivious policy is EAS only in case 1.
  • Figure 4: Path of $\theta^{\textsf{\footnotesize t}}$ as a function of $\varepsilon$ with $\kappa = 20$: a continuous path satisfying $\theta^{\textsf{\footnotesize t}}_{\kappa,\varepsilon} \to \theta^\star$ as $\varepsilon\uparrow 1$.
  • Figure 5: Shown on the left hand side are the trajectories of Q-learning from 500 independent experiments to obtain estimates $\{{\hat{\theta}}^i : 1\le i\le M\}$. The histogram on the right hand side shows estimates of the discounted cost from the $Q^{{\hat{\theta}}^i}$-greedy policy for each parameter.
  • ...and 1 more figures

Theorems & Definitions (6)

  • Lemma 2.1
  • Proposition 2.2
  • Proposition 2.3
  • Proposition 2.4
  • Proposition 2.5
  • Proposition 3.1