PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates

TL;DR

PROMISE presents a framework for scalable preconditioning in stochastic optimization by combining sketch-based Hessian approximations with lazy preconditioner updates. It formalizes three preconditioners—SSN, NySSN, and SASSN—and develops three algorithms (SketchySVRG, SketchySAGA, SketchyKatyusha) that achieve global linear convergence with default hyperparameters, independent of conditioning in ridge problems. The key theoretical innovations are quadratic regularity and Hessian dissimilarity, which yield refined convergence rates and reveal practical gradient-batchsize requirements for effective preconditioning. Empirically, PROMISE methods consistently outperform tuned stochastic optimizers on a large suite of GLMs, including ridge and l2-regularized logistic regression, and scale well to streaming and large-scale datasets, highlighting the practical impact of robust, out-of-the-box optimization in ill-conditioned settings.

Abstract

This paper introduces PROMISE ($\textbf{Pr}$econditioned Stochastic $\textbf{O}$ptimization $\textbf{M}$ethods by $\textbf{I}$ncorporating $\textbf{S}$calable Curvature $\textbf{E}$stimates), a suite of sketching-based preconditioned stochastic gradient algorithms for solving large-scale convex optimization problems arising in machine learning. PROMISE includes preconditioned versions of SVRG, SAGA, and Katyusha; each algorithm comes with a strong theoretical analysis and effective default hyperparameter values. In contrast, traditional stochastic gradient methods require careful hyperparameter tuning to succeed, and degrade in the presence of ill-conditioning, a ubiquitous phenomenon in machine learning. Empirically, we verify the superiority of the proposed algorithms by showing that, using default hyperparameter values, they outperform or match popular tuned stochastic gradient optimizers on a test bed of $51$ ridge and logistic regression problems assembled from benchmark machine learning repositories. On the theoretical side, this paper introduces the notion of quadratic regularity in order to establish linear convergence of all proposed methods even when the preconditioner is updated infrequently. The speed of linear convergence is determined by the quadratic regularity ratio, which often provides a tighter bound on the convergence rate compared to the condition number, both in theory and in practice, and explains the fast global linear convergence of the proposed methods.

Figures (9)

• Figure 1: SketchyKatyusha (an algorithm in the PROMISE suite, see \ref{['alg:skkat']}) with its default hyperparameters outperforms standard stochastic gradient optimizers with both default (left) and tuned (right) hyperparameters. The loss curves start after a single epoch of training has been completed; the black dotted line indicates the training loss attained by SketchyKatyusha after a single epoch. Each optimizer is allotted $1$ hour of runtime.
• Figure 2: PROMISE methods solve ridge regression problems faster than competitors.
• Figure 3: PROMISE methods (and SLBFGS) solve $l^2$-regularized logistic regression problems faster than competitors.
• Figure 4: Suboptimality comparisons between our proposed methods and tuned competitor methods for selected datasets on ridge regression.
• Figure 5: Suboptimality comparisons between our proposed methods and tuned competitor methods for selected datasets on $l^2$-regularized logistic regression.

Theorems & Definitions (31)

• Lemma 1
• Lemma 2
• Lemma 3
• Definition 4: $\zeta$-spectral approximation
• Definition 5: Ridge leverage scores
• Definition 6: Effective dimension and ridge leverage coherence
• Lemma 7: Effective dimension under polynomial decay
• Proposition 8
• Lemma 9: $d^{\nu}_{\textup{eff}}$ vs. $\kappa_{\textup{max}}$ for GLMs
• Proposition 10