A General-Purpose Theorem for High-Probability Bounds of Stochastic Approximation with Polyak Averaging
Sajad Khodadadian, Martin Zubeldia
TL;DR
The paper addresses the lack of non-asymptotic, high-probability bounds for Polyak–Ruppert averaged stochastic approximation in general settings. It develops a general framework that converts high-probability bounds on unaveraged SA iterates into sharp finite-time bounds for the averaged sequence, yielding a leading 1/(k+1) convergence rate and explicit tail terms. A tightness result demonstrates that the leading term is optimal up to a universal constant, and the framework is applied to contractive SA and several reinforcement learning algorithms, including averaged TD-learning, Q-learning, and off-policy TD-learning, deriving new high-probability bounds in settings where prior analyses were limited. The results guide step-size selection and provide practically meaningful concentration guarantees, enhancing reliability for SA-based optimization and RL methods. Overall, the framework offers a modular, broadly applicable approach to quantify the distributional behavior of averaged SA iterates in finite time.
Abstract
Polyak-Ruppert averaging is a widely used technique to achieve the optimal asymptotic variance of stochastic approximation (SA) algorithms, yet its high-probability performance guarantees remain underexplored in general settings. In this paper, we present a general framework for establishing non-asymptotic concentration bounds for the error of averaged SA iterates. Our approach assumes access to individual concentration bounds for the unaveraged iterates and yields a sharp bound on the averaged iterates. We also construct an example, showing the tightness of our result up to constant multiplicative factors. As direct applications, we derive tight concentration bounds for contractive SA algorithms and for algorithms such as temporal difference learning and Q-learning with averaging, obtaining new bounds in settings where traditional analysis is challenging.