Closing the gap between SVRG and TD-SVRG with Gradient Splitting

TL;DR

This work presents TD-SVRG, a variance-reduced TD learning method that interprets TD updates as gradient splitting of a quadratic objective to fuse TD with SVRG. By fixing a learning rate of $1/8$ and leveraging a gradient-splitting framework, the authors establish geometric convergence in both finite-sample and online settings, with a linear dependence on the condition number $\lambda_A$, matching results known for SVRG in convex optimization. They address both i.i.d. and Markovian sampling, and extend the analysis to batching and online estimation of the mean-path update, yielding practical batch sizes and improved complexities over prior work. Experiments across synthetic and OpenAI Gym tasks corroborate the theoretical gains, showing substantial speedups over previous TD-SVRG variants and related variance-reduction TD methods. Overall, the paper tightens the connection between TD learning and convex SVRG theory, delivering faster, scalable policy evaluation in large-scale RL problems.

Abstract

Temporal difference (TD) learning is a policy evaluation in reinforcement learning whose performance can be enhanced by variance reduction methods. Recently, multiple works have sought to fuse TD learning with Stochastic Variance Reduced Gradient (SVRG) method to achieve a geometric rate of convergence. However, the resulting convergence rate is significantly weaker than what is achieved by SVRG in the setting of convex optimization. In this work we utilize a recent interpretation of TD-learning as the splitting of the gradient of an appropriately chosen function, thus simplifying the algorithm and fusing TD with SVRG. Our main result is a geometric convergence bound with predetermined learning rate of $1/8$, which is identical to the convergence bound available for SVRG in the convex setting. Our theoretical findings are supported by a set of experiments.

Figures (11)

• Figure 1: Illustration of gradient splitting. All gradient splittings of the function $f(\theta)$ will lie on line $l$. In addition, if we have a constraint on the 2-norm of the matrix $A$, all gradient splittings will lie on an interval $I$, thus suggesting that an update in the direction of gradient splitting is almost as good, is an update in the direction of the true gradient.
• Figure 2: Geometric average performance of different algorithms in the finite sample case. Columns - dataset source environments: MDP, Acrobot, CartPole and Mountain Car. Rows - performance measurements: $\log(f(\theta) )$ and $\log(|\theta - \theta^*|)$.
• Figure 3: Theoretical batch sizes of different algorithms in log-scale, geometrical average over 10 samples. The $x$-axis plots the dimension of the feature vector. First row: Batch sizes for random MDP environment (see Sec. \ref{['sec_alg_comp']}). Left to right: Figure 1 - 50 states, 20 actions and $\gamma = 0.8$; Figure 2: 400 states, 10 actions and $\gamma = 0.95$, Figure 3: 1000 states, 20 actions and $\gamma = 0.99$; Figure 4: 2000 states, 50 actions and $\gamma = 0.75$. Second row: batch sizes for dataset generated from OpenAI gym classic control environments openai. Features generated by applying RBF kernels and then removing highly correlated feature vectors one by one (see Sec. \ref{['sec_alg_comp']}).
• Figure 4: Average performance of TD-SVRG and PD-SVRG algorithms with different parameters: "SVRG_theory" is TD-SVRG algorithm with parameters suggested by theoretical analysis; "SVRG_lr_1/4_batch_scale_8" is a best-performing algorithm from TD-SVRG search grid ($\alpha = 1/4$, $M = 8/\lambda_A$); "PD-SVRG_0.01_1e-6" is a the best perforing algorithm from the first PD-SVRG search grid ($\sigma_\theta=10^{-6}\frac{1}{L_\rho \kappa (\hat{C})}$, $\sigma_w = 10^{-2} \frac{1}{\lambda_{\rm max}(C)}$); "PD-SCRG_0.01_0.125_8" is the best performing PS-SVRG algorithms from the second grid search ($\alpha = 1/8$, $M = 8/\lambda_A$, $\sigma_w = 10^{-2} \frac{1}{\lambda_{\rm max}(C)}$). Rows - performance measurements: $\log(f(\theta))$ and $\log(|\theta - \theta^*|)$.
• Figure 5: Geometric average performance of different algorithms in the finite sample case with DQN features. Columns - dataset source environments: Acrobot, CartPole and Mountain Car. Rows - performance measurements: $\log(f(\theta) )$ and $\log(|\theta - \theta^*|)$.

Theorems & Definitions (15)

• Lemma 4.1
• proof
• Theorem 5.1
• Corollary 5.2
• proof : Proof of Theorem \ref{['mainthm1']}
• Theorem 5.3
• proof
• Theorem 5.4
• proof
• Theorem 5.6