Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Newton Proximal Extragradient Method

Ruichen Jiang, Michał Dereziński, Aryan Mokhtari

TL;DR

A novel stochastic Newton proximal extragradient method is proposed that improves these bounds, achieving a faster global linear rate and reaching the same fast superlinear rate in $\tilde{O}(\kappa)$ iterations.

Abstract

Stochastic second-order methods achieve fast local convergence in strongly convex optimization by using noisy Hessian estimates to precondition the gradient. However, these methods typically reach superlinear convergence only when the stochastic Hessian noise diminishes, increasing per-iteration costs over time. Recent work in [arXiv:2204.09266] addressed this with a Hessian averaging scheme that achieves superlinear convergence without higher per-iteration costs. Nonetheless, the method has slow global convergence, requiring up to $\tilde{O}(κ^2)$ iterations to reach the superlinear rate of $\tilde{O}((1/t)^{t/2})$, where $κ$ is the problem's condition number. In this paper, we propose a novel stochastic Newton proximal extragradient method that improves these bounds, achieving a faster global linear rate and reaching the same fast superlinear rate in $\tilde{O}(κ)$ iterations. We accomplish this by extending the Hybrid Proximal Extragradient (HPE) framework, achieving fast global and local convergence rates for strongly convex functions with access to a noisy Hessian oracle.

Stochastic Newton Proximal Extragradient Method

TL;DR

A novel stochastic Newton proximal extragradient method is proposed that improves these bounds, achieving a faster global linear rate and reaching the same fast superlinear rate in iterations.

Abstract

Stochastic second-order methods achieve fast local convergence in strongly convex optimization by using noisy Hessian estimates to precondition the gradient. However, these methods typically reach superlinear convergence only when the stochastic Hessian noise diminishes, increasing per-iteration costs over time. Recent work in [arXiv:2204.09266] addressed this with a Hessian averaging scheme that achieves superlinear convergence without higher per-iteration costs. Nonetheless, the method has slow global convergence, requiring up to iterations to reach the superlinear rate of , where is the problem's condition number. In this paper, we propose a novel stochastic Newton proximal extragradient method that improves these bounds, achieving a faster global linear rate and reaching the same fast superlinear rate in iterations. We accomplish this by extending the Hybrid Proximal Extragradient (HPE) framework, achieving fast global and local convergence rates for strongly convex functions with access to a noisy Hessian oracle.
Paper Structure (29 sections, 30 theorems, 121 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 29 sections, 30 theorems, 121 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Stochastic Newton Proximal Extragradient
  4. Key properties of SNPE
  5. Analysis of uniform Hessian averaging
  6. Convergence analysis
  7. Analysis of weighted Hessian averaging
  8. Discussion and complexity comparison
  9. Numerical experiments
  10. Conclusions and limitations
  11. Missing proofs in Section \ref{['sec:stochastic_NPE']}
  12. Proof of Proposition \ref{['prop:HPE']}
  13. Proof of Lemma \ref{['lem:step_size_bnd']}
  14. Extension to inexact linear solving
  15. The total complexity of line search
  16. ...and 14 more sections

Key Result

Proposition 1

Let $\{{\mathbf{x}}_t\}_{t\geq 0}$ and $\{\hat{{\mathbf{x}}}_t\}_{t\geq 0}$ be the iterates generated by Algorithm alg:stochastic_NPE. Then for any $t \geq 0$, we have $\|{\mathbf{x}}_{t+1} - {\mathbf{x}}^*\|^2 \leq \|{\mathbf{x}}_t-{\mathbf{x}}^*\|^2(1+2\eta_t \mu)^{-1}$.

Figures (4)

  • Figure 1: Stochastic NPE
  • Figure 2: Iteration complexity comparison for minimizing log-sum-exp on a synthetic dataset.
  • Figure 3: Runtime comparison for minimizing log-sum-exp on a synthetic dataset.
  • Figure 4: The effect of the extragradient step in stochastic NPE.

Theorems & Definitions (59)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Proposition 1
  • Lemma 1
  • Lemma 2
  • Lemma 3: na2022hessian
  • Remark 5
  • Lemma 4
  • ...and 49 more