On the Convergence of HalpernSGD
Vittorio Colao, Katherine Rossella Foglia
TL;DR
The paper analyzes HalpernSGD, an anchored stochastic gradient method that combines Halpern fixed-point iterations with a stochastic gradient step for convex $L$-smooth objectives. By coupling the Halpern weights $\{\alpha_n\}$ with diminishing steps $\{\epsilon_n\}$ and assuming martingale-difference noise with bounded variance, it proves almost sure convergence of the iterates to the anchor-selected minimizer $x^*=P_S(u)$, along with $L^2$-boundedness and gradient-norm convergence. Central to the argument are a Xu-type recurrence, boundedness lemmas, and a set of standard stochastic-approximation conditions, including $\epsilon_n\to0$, $\sum \epsilon_n=\infty$, $\sum \epsilon_n^2<\infty$, $\alpha_n\to0$, $\sum \alpha_n=\infty$, and $\sum \alpha_n^2<\infty$, plus a coupling $\lim \epsilon_n/\alpha_n=\infty$ and $(1-\alpha_n)+\epsilon_n^2 L^2\le1$. The results fill a theoretical gap for HalpernSGD and pave the way for anchored adaptations of adaptive methods like Adam.
Abstract
HalpernSGD is a gradient-type optimizer obtained by combining the classical Halpern fixed-point iteration with a stochastic gradient step. While its empirical advantages have already been observed in \cite{colao2025optimizer,foglia2024halpernsgd}, this paper provides a theoretical analysis of the method. Assuming a convex $L$-smooth objective and standard stochastic-approximation conditions, we prove almost sure convergence of the iterates to a minimizer, characterized as the metric projection of the anchor onto the solution set. These results fill a gap in the literature on HalpernSGD and lay the groundwork for future extensions to adaptive schemes.