Regularized Gradient Temporal-Difference Learning

TL;DR

The paper tackles instability in off-policy policy evaluation with linear function approximation caused by singular feature interaction in GTD2. It introduces Regularized GTD (R-GTD), which regularizes MSPBE via a slack variable and a quadratic penalty in a convex–concave saddle-point framework, yielding a unique solution without requiring FIM nonsingularity. The authors prove global convergence to the saddle point using PDGD/ODE methods and derive explicit error bounds to the true projected value, clarifying behavior in both singular and nonsingular cases and showing convergence to GTD2 as the regularization grows. Empirical results in singular settings demonstrate improved stability and accuracy of R-GTD over GTD2, with larger regularization yielding further gains, while appendices extend validation to additional scenarios such as Baird’s counterexample and stochastic environments.

Abstract

Gradient temporal-difference (GTD) learning algorithms are widely used for off-policy policy evaluation with function approximation. However, existing convergence analyses rely on the restrictive assumption that the so-called feature interaction matrix (FIM) is nonsingular. In practice, the FIM can become singular and leads to instability or degraded performance. In this paper, we propose a regularized optimization objective by reformulating the mean-square projected Bellman error (MSPBE) minimization. This formulation naturally yields a regularized GTD algorithms, referred to as R-GTD, which guarantees convergence to a unique solution even when the FIM is singular. We establish theoretical convergence guarantees and explicit error bounds for the proposed method, and validate its effectiveness through empirical experiments.

Figures (8)

• Figure 1: As $c \to \infty$, the R-GTD solution $\theta_{\mathrm{RGTD}}$ converges to the GTD2 solution $\theta_{\mathrm{GTD2}}$. $\theta_{\mathrm{GTD2}}$ decomposes uniquely into two components: $v \in \mathrm{Null}(G)$ along the null space of $G$, and $v_{\perp} \in \mathrm{Null}(G)^{\perp}$ orthogonal to it.
• Figure 2: Solution trajectory of the closed-form $\theta_{\mathrm{RGTD}}$ in a two-dimensional singular case toy example. As the regularization parameter $c$ increases, the $\theta_{\mathrm{RGTD}}$ converges to the $\theta_{\mathrm{GTD2}}$.
• Figure 3: Geometric illustration of the singular case. The error bounds of R-GTD and GTD2 are compared through the projection and distance terms in \ref{['thm:2']}.
• Figure 4: (a) Convergence of $\|\theta_k - \theta_*^\pi\|_2$ for GTD2 and R-GTD with step-size $\alpha_k = 1/(k + 30)$ and regularization coefficients $c_1=0.2$, $c_2=0.4$, and $c_3=1$. (b) Box-and-whisker representation of $\|\theta_k - \theta_*^\pi\|_2$ over 30 independent runs corresponding to (a).
• Figure 5: (a) Convergence of $\|\theta_k - \theta_*^\pi\|_2$ for GTD2 and R-GTD with step-size $\alpha_k = 1/(k + 30)$ and regularization coefficients $c_1 = 0.2$, $c_2 = 0.4$, and $c_3 = 1$. (b) Box-and-whisker representation of $\|\theta_k - \theta_*^\pi\|_2$ over 30 runs across three MDPs, showing the median, interquartile range, and outliers.

Theorems & Definitions (25)

• Definition 2.1: Saddle point
• Proposition 4.1
• Theorem 5.1
• Lemma 5.2
• Lemma 5.3
• Theorem 5.4
• Lemma 1.1
• Lemma 1.3: Borkar and Meyn theorem, borkar2000ode
• Lemma 1.4
• Lemma 1.5