Regularized Q-learning

TL;DR

This work tackles instability of Q-learning with linear function approximation under bootstrapping and off-policy by introducing Regularized Q-learning (RegQ), a single-time-scale algorithm that adds an $l_2$ penalty to ensure convergence. The authors formulate a regularized projected Bellman equation (RPBE) with a regularized projection $\\Gamma_{\\eta}$ and prove existence, uniqueness, and convergence of the RegQ iterates to $\\theta^*_{\\eta}$ via an ODE-based switching-system analysis and Borkar–Meyn theory. They derive an explicit error bound between the RPBE solution and the true Q-function, describe regimes where this bias vanishes or persists, and provide conditions on the regularization parameter $\\eta$ to guarantee convergence. Empirically, RegQ stabilizes learning in classic divergence-prone settings (e.g., Baird counterexample) and demonstrates faster convergence than prior two-time-scale methods, suggesting practical stability without target networks and potential for extension to deeper RL with preconditioning rather than target networks.

Abstract

Q-learning is widely used algorithm in reinforcement learning community. Under the lookup table setting, its convergence is well established. However, its behavior is known to be unstable with the linear function approximation case. This paper develops a new Q-learning algorithm that converges when linear function approximation is used. We prove that simply adding an appropriate regularization term ensures convergence of the algorithm. We prove its stability using a recent analysis tool based on switching system models. Moreover, we experimentally show that it converges in environments where Q-learning with linear function approximation has known to diverge. We also provide an error bound on the solution where the algorithm converges.

Figures (6)

• Figure 1: Illustrative explanation on the regularized projection. The Figure \ref{['fig:e3']} implies that as $\eta\to\infty$, $\gamma \Gamma_\eta$ can potentially move outside of the unit ball satisfying $||x||_{\infty}\le 1$, and this phase is indicated with the term "blowing up" phase. The quantity $\| \gamma \Gamma_\eta \|_\infty$ actually blows up initially as $\eta\to \infty$. However, since $\lim_{\eta\to\infty } {\| \gamma \Gamma _\eta \|_\infty } = 0$, we know that $\gamma \Gamma_\eta$ will eventually converge to the origin and move inside the unit ball. This behavior is indicated by the "shrinking" phase in the figure.
• Figure 2: Experiment results
• Figure 3: State transition diagram
• Figure 4: Counter-examples where Q-learning with linear function approximation diverges
• Figure 5: Learning curve under different learning rate and regularization coefficient

Theorems & Definitions (45)

• Lemma 2.4: gosavi2006boundedness
• Lemma 2.5
• Lemma 3.1
• Lemma 3.2
• Lemma 3.3
• Lemma 3.4
• Lemma 3.5
• Lemma 3.6
• Lemma 3.7
• Theorem 5.1