A Piecewise Lyapunov Analysis of Sub-quadratic SGD: Applications to Robust and Quantile Regression

TL;DR

This work develops a rigorous analysis for SGD applied to objectives that are locally strongly convex but have sub-quadratic tails, introducing a novel piecewise Lyapunov function to accommodate first-order differentiable losses such as the Huber loss. The authors establish finite-time moment bounds for both constant and diminishing stepsizes, and prove weak convergence, a central limit theorem, and an asymptotic bias characterization under constant stepsize, achieving the first geometric convergence results for sub-quadratic SGD. They then apply the theory to online robust regression and online quantile regression, deriving explicit constants, relaxing prior assumptions (e.g., density continuity in quantile regression), and obtaining rates that align with or reach CRLB-type benchmarks under appropriate conditions. The results provide practically relevant guidance for online statistical methods with heavy-tailed data or corruption, including implications for bias reduction via Richardson–Romberg extrapolation and efficient averaging strategies. Overall, the paper delivers a unified, sharp framework for sub-quadratic SGD with broad online regression applications and opens avenues for extension to sub-linear regimes and broader loss classes.

Abstract

Motivated by robust and quantile regression problems, we investigate the stochastic gradient descent (SGD) algorithm for minimizing an objective function $f$ that is locally strongly convex with a sub--quadratic tail. This setting covers many widely used online statistical methods. We introduce a novel piecewise Lyapunov function that enables us to handle functions $f$ with only first-order differentiability, which includes a wide range of popular loss functions such as Huber loss. Leveraging our proposed Lyapunov function, we derive finite-time moment bounds under general diminishing stepsizes, as well as constant stepsizes. We further establish the weak convergence, central limit theorem and bias characterization under constant stepsize, providing the first geometrical convergence result for sub--quadratic SGD. Our results have wide applications, especially in online statistical methods. In particular, we discuss two applications of our results. 1) Online robust regression: We consider a corrupted linear model with sub--exponential covariates and heavy--tailed noise. Our analysis provides convergence rates comparable to those for corrupted models with Gaussian covariates and noise. 2) Online quantile regression: Importantly, our results relax the common assumption in prior work that the conditional density is continuous and provide a more fine-grained analysis for the moment bounds.

Figures (4)

• Figure 1: Convergence with different diminishing stepsizes and the convergence rate
• Figure 2: Asymptotic Normality for online robust regression
• Figure 3: Error of TA and RR-extrapolated iterates using constant stepsize comparing with the error of raw iterates using diminishing stepsize for online robust regression
• Figure 4: Online robust regression and quantile regression with different model parameters

Theorems & Definitions (23)

• Definition 1: $\psi_q-$Orlicz Space
• Lemma 1
• Proposition 1
• Proposition 2
• Theorem 1: Moment Bounds with Constant Stepsize
• Theorem 2: Moment Bounds with Diminishing Stepsize
• Theorem 3: Weak Convergence
• Theorem 4: Central Limit Theorem
• Corollary 1: Bias Characterizaion
• Theorem 5