Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variance Reduction Based Experience Replay for Policy Optimization

Hua Zheng, Wei Xie, M. Ben Feng, Keilung Choy

TL;DR

This paper tackles low sample efficiency in policy-gradient RL by proposing Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative step-based experiences to reduce gradient variance. VRER integrates with existing policy optimization methods to produce PG-VRER, and the authors provide a novel finite-time convergence analysis that accounts for Markovian noise and behavior-policy interdependence, revealing a bias-variance trade-off in reusing older samples. The framework relies on a variance-based sample selection rule, bias-mitigating downsampling, and a moving-block bootstrap variance estimator, together enabling practical, off-policy improvements with theoretical guarantees. Empirically, VRER consistently accelerates convergence and improves performance across standard Gym/PyBullet benchmarks for TRPO, PPO, and A2C variants, while analyzing reuse patterns and parameter sensitivity to guide practical deployment.

Abstract

Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data collected in previous iterations to accelerate policy optimization. Classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To overcome this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce variance in policy gradient estimation. VRER is algorithm-agnostic and integrates seamlessly with existing policy optimization methods, forming the basis of our sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). Motivated by the lack of rigorous theoretical analysis of experience replay, we develop a novel framework that explicitly captures dependencies introduced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and reveal a fundamental bias-variance trade-off: reusing older experience increases bias but simultaneously reduces gradient variance. Extensive empirical experiments demonstrate that VRER consistently accelerates policy learning and improves performance over state-of-the-art policy optimization algorithms.

Variance Reduction Based Experience Replay for Policy Optimization

TL;DR

This paper tackles low sample efficiency in policy-gradient RL by proposing Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative step-based experiences to reduce gradient variance. VRER integrates with existing policy optimization methods to produce PG-VRER, and the authors provide a novel finite-time convergence analysis that accounts for Markovian noise and behavior-policy interdependence, revealing a bias-variance trade-off in reusing older samples. The framework relies on a variance-based sample selection rule, bias-mitigating downsampling, and a moving-block bootstrap variance estimator, together enabling practical, off-policy improvements with theoretical guarantees. Empirically, VRER consistently accelerates convergence and improves performance across standard Gym/PyBullet benchmarks for TRPO, PPO, and A2C variants, while analyzing reuse patterns and parameter sensitivity to guide practical deployment.

Abstract

Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data collected in previous iterations to accelerate policy optimization. Classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To overcome this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce variance in policy gradient estimation. VRER is algorithm-agnostic and integrates seamlessly with existing policy optimization methods, forming the basis of our sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). Motivated by the lack of rigorous theoretical analysis of experience replay, we develop a novel framework that explicitly captures dependencies introduced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and reveal a fundamental bias-variance trade-off: reusing older experience increases bias but simultaneously reduces gradient variance. Extensive empirical experiments demonstrate that VRER consistently accelerates policy learning and improves performance over state-of-the-art policy optimization algorithms.
Paper Structure (29 sections, 8 theorems, 61 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 29 sections, 8 theorems, 61 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Description
  4. Regularity Conditions for Policy Gradient Estimation
  5. Likelihood Ratio Based Policy Gradient Estimation
  6. Step-based Policy Optimization via Importance Sampling
  7. Likelihood Ratio (LR) Policy Gradient Estimator
  8. Clipped Likelihood Ratio (CLR) Policy Gradient Estimator
  9. Variance Reduction Replay for Policy Optimization
  10. Selection Rule for Reducing Policy Gradient Variance
  11. Variance Estimation for Policy Gradient Estimators
  12. Gradient Variance Ratio Approximation
  13. VRER Assisted Generic Policy Optimization Algorithm
  14. Finite-Time Convergence Analysis of PG-VRER
  15. Empirical Study
  16. ...and 14 more sections

Key Result

Lemma 1

Under Assumption assumption 2, the policy gradient of objective, denoted by $\nabla J(\pmb\theta)$, is Lipschitz continuous, i.e., for any policy parameters $\pmb\theta_1,\pmb\theta_2 \in\Theta$, there exists a constant $L > 0$ s.t.

Figures (4)

  • Figure 1: Convergence results for the various PG algorithms with and without using VRER.
  • Figure 2: Sensitivity analysis of buffer sizes. The average rewards achieved by PPO-VRER, TRPO-VRER, and A2C-VRER in the CartPole task with different buffer sizes.
  • Figure 3: Sensitivity of the Selection Constant. The average rewards achieved by PPO-VRER, TRPO-VRER, and A2C-VRER in the CartPole task with different $c$.
  • Figure 4: Comparison between A2C-VRER and ACER approach in discrete control Tasks.

Theorems & Definitions (11)

  • Lemma 1: zhang2020global, Lemma 3.2
  • Lemma 2: Boundedness of Stochastic Policy Gradients
  • Theorem 1
  • Proposition 2
  • Remark 1: Bias Reduction
  • Remark 2: Buffer Size
  • Remark 3
  • Theorem 3: Convergence of PG-VRER
  • Corollary 1
  • Corollary 2
  • ...and 1 more