Extremely Fast Convergence Rates for Extremum Seeking Control with Polyak-Ruppert Averaging

TL;DR

It is shown in this paper that through design it is possible to obtain far faster convergence, of order $O(n^{-4+\delta})$, with $\delta>0$ arbitrary.

Abstract

Stochastic approximation is a foundation for many algorithms found in machine learning and optimization. It is in general slow to converge: the mean square error vanishes as $O(n^{-1})$. A deterministic counterpart known as quasi-stochastic approximation is a viable alternative in many applications, including gradient-free optimization and reinforcement learning. It was assumed in prior research that the optimal achievable convergence rate is $O(n^{-2})$. It is shown in this paper that through design it is possible to obtain far faster convergence, of order $O(n^{-4+δ})$, with $δ>0$ arbitrary. Two techniques are introduced for the first time to achieve this rate of convergence. The theory is also specialized within the context of gradient-free optimization, and tested on standard benchmarks. The main results are based on a combination of novel application of results from number theory and techniques adapted from stochastic approximation theory.

Figures (14)

• Figure 1: What is the optimal convergence rate for QSA?
• Figure 2: Rates of convergence for PR averaging and forward-backward algorithms.
• Figure 3: Quasi Monte Carlo using QSA with Polyak-Ruppert averaging. Histograms for $n=10\times T =10^5$.
• Figure 4: Hidden geometry: orthogonality of the function classes $S$ and ${\widehat{S}}$.
• Figure 5: Comparison of Monte Carlo and quasi-Monte Carlo with Polyak-Ruppert averaging.

Theorems & Definitions (49)

• Theorem 2.1
• Theorem 2.2
• Theorem 2.3
• Proposition 3.1
• Corollary 3.2
• Proposition A.1
• Theorem B.1
• Proposition B.2
• proof
• Lemma B.3