On the Convergence of Experience Replay in Policy Optimization: Characterizing Bias, Variance, and Finite-Time Convergence
Hua Zheng, Wei Xie, M. Ben Feng
TL;DR
The paper addresses the theoretical gaps in understanding experience replay for policy gradient methods by introducing an auxiliary Markov-chain framework with lag-based decoupling to disentangle Markov noise and policy drift. It derives finite-time bias bounds for LR/CLR estimators that depend on cumulative policy updates, mixing rates, and replay age, and provides a correlation-aware variance decomposition showing when replay reduces variance. Building on these characterizations, it establishes finite-time convergence guarantees for ER-based policy optimization, revealing a fundamental bias-variance trade-off: larger buffers offer variance reduction but increase data staleness bias. The results yield principled guidance for buffer sizing, replay schedules, and lag choices, connecting empirical heuristics with rigorous quantitative theory and highlighting the central role of environment mixing in ER effectiveness.
Abstract
Experience replay is a core ingredient of modern deep reinforcement learning, yet its benefits in policy optimization are poorly understood beyond empirical heuristics. This paper develops a novel theoretical framework for experience replay in modern policy gradient methods, where two sources of dependence fundamentally complicate analysis: Markovian correlations along trajectories and policy drift across optimization iterations. We introduce a new proof technique based on auxiliary Markov chains and lag-based decoupling that makes these dependencies tractable. Within this framework, we derive finite-time bias bounds for policy-gradient estimators under replay, identifying how bias scales with the cumulative policy update, the mixing time of the underlying dynamics, and the age of buffered data, thereby formalizing the practitioner's rule of avoiding overly stale replay. We further provide a correlation-aware variance decomposition showing how sample dependence governs gradient variance from replay and when replay is beneficial. Building on these characterizations, we establish the finite-time convergence guarantees for experience-replay-based policy optimization, explicitly quantifying how buffer size, sample correlation, and mixing jointly determine the convergence rate and revealing an inherent bias-variance trade-off: larger buffers can reduce variance by averaging less correlated samples but can increase bias as data become stale. These results offer a principled guide for buffer sizing and replay schedules, bridging prior empirical findings with quantitative theory.