Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

Andre Milzarek, Fabian Schaipp, Michael Ulbrich

TL;DR

Numerical experiments show that the proposed stochastic proximal point algorithm competes favorably with other state-of-the-art methods and achieves higher robustness with respect to the step size selection.

Abstract

We develop an implementable stochastic proximal point (SPP) method for a class of weakly convex, composite optimization problems. The proposed stochastic proximal point algorithm incorporates a variance reduction mechanism and the resulting SPP updates are solved using an inexact semismooth Newton framework. We establish detailed convergence results that take the inexactness of the SPP steps into account and that are in accordance with existing convergence guarantees of (proximal) stochastic variance-reduced gradient methods. Numerical experiments show that the proposed algorithm competes favorably with other state-of-the-art methods and achieves higher robustness with respect to the step size selection.

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

TL;DR

Numerical experiments show that the proposed stochastic proximal point algorithm competes favorably with other state-of-the-art methods and achieves higher robustness with respect to the step size selection.

Abstract

We develop an implementable stochastic proximal point (SPP) method for a class of weakly convex, composite optimization problems. The proposed stochastic proximal point algorithm incorporates a variance reduction mechanism and the resulting SPP updates are solved using an inexact semismooth Newton framework. We establish detailed convergence results that take the inexactness of the SPP steps into account and that are in accordance with existing convergence guarantees of (proximal) stochastic variance-reduced gradient methods. Numerical experiments show that the proposed algorithm competes favorably with other state-of-the-art methods and achieves higher robustness with respect to the step size selection.
Paper Structure (30 sections, 14 theorems, 105 equations, 7 figures, 2 tables, 2 algorithms)

This paper contains 30 sections, 14 theorems, 105 equations, 7 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Contributions
  4. Preliminaries
  5. The Stochastic Proximal Point Method
  6. Assumptions
  7. Algorithmic Framework
  8. A Semismooth Newton Method for Solving the Subproblem
  9. Controlling the Inexactness of the Update
  10. Convergence Analysis
  11. Weakly Convex Case
  12. Strongly Convex Case
  13. Numerical Experiments
  14. General Setting
  15. Logistic Regression with $\ell_1$-Regularization
  16. ...and 15 more sections

Key Result

Proposition 2.2

\newlabelprop:strongly_convex_lipschitz0 Let $g: \mathbb{R}^n \to (-\infty,\infty]$ be proper and closed. If $g$ is $\mu$-strongly convex, then its conjugate $g^\ast$ is closed, convex, proper, and Fréchet differentiable and its gradient is given by $\nabla g^\ast(x) = \mathop{\mathrm{argmax}}\lim

Figures (7)

  • Figure 1: Sparse logistic regression for news20. Left: Objective function gap for different batch and step sizes. Top Right: average runtime for solving the subproblem once (with \ref{['alg:semismooth_newton']}) and for computing $\nabla f(\tilde{x})$. Displayed for $\alpha=1000,b=0.005\cdot N$. Bottom Right: Mean runtime (per iteration index $k$ in SNSPP) for solving the subproblem.
  • Figure 1: Objective function convergence for the logistic regression datasets with respect to number of gradient evaluations. All settings are identical to \ref{['fig:logreg_obj']}.
  • Figure 2: Runtime until convergence for different choices of step and batch sizes.
  • Figure 2: Convergence plot for logistic regression on the madelon.2 dataset, with respect to runtime (left) and number of gradient evaluations (right).
  • Figure 3: Objective function convergence for the logistic regression datasets. For SAGA and AdaGrad, one marker denotes one epoch. For SVRG one marker denotes one outer-loop iteration while for SNSPP it denotes one (inner-loop) iteration.
  • ...and 2 more figures

Theorems & Definitions (31)

  • Definition 2.1
  • Proposition 2.2
  • Proof 1
  • Definition 2.3
  • Definition 3.4
  • Remark 3.5
  • Proposition 4.1
  • Proposition 4.2
  • Proof 2
  • Proposition 4.3
  • ...and 21 more