Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Periodic Regularized Q-Learning

Hyukjun Yang, Han-Dong Lim, Donghwan Lee

TL;DR

This work introduces regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm and provides a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.

Abstract

In reinforcement learning (RL), Q-learning is a fundamental algorithm whose convergence is guaranteed in the tabular setting. However, this convergence guarantee does not hold under linear function approximation. To overcome this limitation, a significant line of research has introduced regularization techniques to ensure stable convergence under function approximation. In this work, we propose a new algorithm, periodic regularized Q-learning (PRQ). We first introduce regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm. By appropriately regularizing the projection operator, the resulting projected value iteration becomes a contraction. By extending this regularized projection into the stochastic setting, we establish the PRQ algorithm and provide a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.

Periodic Regularized Q-Learning

TL;DR

This work introduces regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm and provides a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.

Abstract

In reinforcement learning (RL), Q-learning is a fundamental algorithm whose convergence is guaranteed in the tabular setting. However, this convergence guarantee does not hold under linear function approximation. To overcome this limitation, a significant line of research has introduced regularization techniques to ensure stable convergence under function approximation. In this work, we propose a new algorithm, periodic regularized Q-learning (PRQ). We first introduce regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm. By appropriately regularizing the projection operator, the resulting projected value iteration becomes a contraction. By extending this regularized projection into the stochastic setting, we establish the PRQ algorithm and provide a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.
Paper Structure (40 sections, 33 theorems, 183 equations, 5 figures, 1 table, 2 algorithms)

This paper contains 40 sections, 33 theorems, 183 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Regularized methods and Bellman equation
  4. Target-based update
  5. Contributions
  6. Preliminaries and notations
  7. Markov decision process
  8. Notations
  9. Projected Bellman equation
  10. Regularized projection operator
  11. Regularized projected value iteration
  12. Periodic regularized Q-learning
  13. Main theoretical result
  14. Outer loop decomposition
  15. i.i.d. observation model
  16. ...and 25 more sections

Key Result

Lemma 3.1

[Lemma 3.1 in limregularized] The matrix ${\bm{\Gamma}}_\eta$ satisfies the following properties: $\mathop {\lim }\limits_{\eta \to \infty } {{\bm{\Gamma}} _\eta } \!=\! 0$ and $\mathop {\lim }\limits_{\eta \to 0} {{\bm{\Gamma}} _\eta } = {\bm{\Gamma}}$.

Figures (5)

  • Figure 1: Illustration of the regularized projection. With a proper choice of $\eta$, ${\bm{\Gamma}}_{\eta}{\mathcal{T}}{\bm{x}}$ and ${\bm{\Gamma}}_{\eta}{\mathcal{T}}{\bm{y}}$ will be close to the origin and $||{\bm{\Gamma}}_{\eta}{\mathcal{T}}({\bm{x}}-{\bm{y}})||_2\leq||{\bm{\Gamma}}_{}{\mathcal{T}}({\bm{x}}-{\bm{y}})||_{2}$.
  • Figure 2: Comparison of PRQ and RegQ in the model-based setting with $\eta=0.01$. The top row corresponds to PRQ, while the bottom row corresponds to RegQ. In each row, the left subplot shows the temporal evolution of the parameters $\theta_0$ and $\theta_1$ during iterations, and the right subplot shows the corresponding trajectory in the two-dimensional $(\theta_0, \theta_1)$ parameter space, including the initialization point and the RP-BE solution. PRQ exhibits stable convergence toward the solution, whereas RegQ displays periodic behavior and fails to show convergence.
  • Figure 3: Comparison of PRQ and RegQ in the i.i.d. sample-based setting with $\eta=0.01$. The figure follows the same layout as \ref{['fig:model-exp']}.
  • Figure 4: Comparison of PRQ and RegQ under the Markovian sample-based setting with $\eta=0.01$. The figure follows the same layout as \ref{['fig:model-exp']}.
  • Figure 5: Double-loop structure of PRQ. The inner loop performs gradient descent to solve the regularized subproblem for K times, and the outer loop updates the target parameter.

Theorems & Definitions (70)

  • Lemma 3.1
  • Lemma 3.2
  • Remark 3.3
  • Remark 3.4
  • Lemma 4.1
  • Proposition 6.1
  • Remark 6.2
  • Lemma 6.3: Strong convexity and smoothness of $L_{\eta}({\bm{\theta}},{\bm{\theta}}^{\prime})$
  • Theorem 6.4
  • Theorem 6.5
  • ...and 60 more