Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization

TL;DR

This work studies the minimization of a convex, $L$-smooth function $f$ on a separable real Hilbert space using reg-SGD, a stochastic gradient method augmented by a time-decaying Tikhonov regularization with parameter $\lambda_k$. By introducing an energy Lyapunov function and balancing step-sizes with regularization decay, the authors prove almost-sure and $L^2$ convergence of the last iterate to the minimum-norm solution $x_\ast$ without requiring a priori boundedness, and they derive convergence rates for polynomial schedules and refinements under the Łojasiewicz condition. The results demonstrate that vanishing regularization stabilizes the SGD flow and enables convergence to the unique minimum-norm solution, with practical validation in image reconstruction and ODE-based inverse problems (e.g., X-ray tomography). The paper provides explicit guidelines for choosing polynomial decay rates $(\alpha_k,\lambda_k)$ to optimize convergence, contributing to the design of robust iterative algorithms for convex and ill-posed problems. Overall, reg-SGD offers stable, provably convergent learning dynamics that extend SGD to settings where minimum-norm selection is desirable.

Abstract

The present article studies the minimization of convex, L-smooth functions defined on a separable real Hilbert space. We analyze regularized stochastic gradient descent (reg-SGD), a variant of stochastic gradient descent that uses a Tikhonov regularization with time-dependent, vanishing regularization parameter. We prove strong convergence of reg-SGD to the minimum-norm solution of the original problem without additional boundedness assumptions. Moreover, we quantify the rate of convergence and optimize the interplay between step-sizes and regularization decay. Our analysis reveals how vanishing Tikhonov regularization controls the flow of SGD and yields stable learning dynamics, offering new insights into the design of iterative algorithms for convex problems, including those that arise in ill-posed inverse problems. We validate our theoretical findings through numerical experiments on image reconstruction and ODE-based inverse problems.

Figures (14)

• Figure 1: A comparison of SGD and reg-SGD, reg-SGD converges to $x_\ast$ for all initializations.
• Figure 2: Optimal choices of $p$ and $q$. Left: convergence rate for $\mathbb{E}[\|X_k-x_\ast\|_\mathcal{X}^2]$ in the situation of \ref{['thm:strongconv_L2']}. Right: almost sure convergence rate for $\|X_k-x_\ast\|_\mathcal{X}^2$ in the situation of \ref{['thm:strongconv_as']} under the Polyak-Ł ojasiewicz inequality.
• Figure 3: Left: base image. Middle: Radon transform. Right: minimum-norm solution $x_\ast$.
• Figure 4: Left: reconstruction using reg-SGD with our optimal schedules. Middle: reconstruction using reg-SGD with more aggressive schedules. Right: reconstruction using vanilla SGD.
• Figure 5: Left: pathwise optimality gap $f(X_k)-f(x_\ast)$. Right: pathwise squared error to the minimum-norm solution $\|X_k-x_\ast\|^2$. Each curve represents one of $10$ independent runs, each of length $N=5\cdot 10^6$. The red shaded lines depict individual runs of SGD, while the green dash-dotted and blue dashed shaded lines correspond to reg-SGD. The red solid line shows the average error across runs for SGD, the green bold dash-dotted and blue dashed line shows the average for reg-SGD, and the black dashed line indicates the theoretical convergence rate.

Theorems & Definitions (37)

• Example 1: Mini-batch estimator for finite-sum problems
• Theorem 2.1: Almost sure convergence
• Theorem 2.2: $L^2$-convergence
• Theorem 2.3: $L^2$-rates for reg-SGD with polynomial schedules
• Theorem 2.4: Almost sure-rates for reg-SGD with polynomial schedules
• Remark 1
• Corollary 1: Strong $L^2$ convergence rates
• Corollary 2: Strong a.s. convergence rates
• Lemma 1
• proof