Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Inexact Stochastic ADMM for Composite Nonconvex and Nonsmooth Optimization

Yuxuan Zeng, Jianchao Bai, Shengjia Wang, Zhiguo Wang

TL;DR

A unified framework of inexact stochastic Alternating Direction Method of Multipliers for solving nonconvex problems subject to linear constraints, whose objective comprises an average of finite-sum smooth functions and a nonsmooth but possibly nonconvex function is proposed.

Abstract

In this paper, we propose a unified framework of inexact stochastic Alternating Direction Method of Multipliers (ADMM) for solving nonconvex problems subject to linear constraints, whose objective comprises an average of finite-sum smooth functions and a nonsmooth but possibly nonconvex function. The new framework is highly versatile. Firstly, it not only covers several existing algorithms such as SADMM, SVRG-ADMM, and SPIDER-ADMM but also guides us to design a novel accelerated hybrid stochastic ADMM algorithm, which utilizes a new hybrid estimator to trade-off variance and bias. Second, it enables us to exploit a more flexible dual stepsize in the convergence analysis. Under some mild conditions, our unified framework preserves $\mathcal{O}(1/T)$ sublinear convergence. Additionally, we establish the linear convergence under error bound conditions. Finally, numerical experiments demonstrate the efficacy of the new algorithm for some nonsmooth and nonconvex problems.

A Unified Inexact Stochastic ADMM for Composite Nonconvex and Nonsmooth Optimization

TL;DR

A unified framework of inexact stochastic Alternating Direction Method of Multipliers for solving nonconvex problems subject to linear constraints, whose objective comprises an average of finite-sum smooth functions and a nonsmooth but possibly nonconvex function is proposed.

Abstract

In this paper, we propose a unified framework of inexact stochastic Alternating Direction Method of Multipliers (ADMM) for solving nonconvex problems subject to linear constraints, whose objective comprises an average of finite-sum smooth functions and a nonsmooth but possibly nonconvex function. The new framework is highly versatile. Firstly, it not only covers several existing algorithms such as SADMM, SVRG-ADMM, and SPIDER-ADMM but also guides us to design a novel accelerated hybrid stochastic ADMM algorithm, which utilizes a new hybrid estimator to trade-off variance and bias. Second, it enables us to exploit a more flexible dual stepsize in the convergence analysis. Under some mild conditions, our unified framework preserves sublinear convergence. Additionally, we establish the linear convergence under error bound conditions. Finally, numerical experiments demonstrate the efficacy of the new algorithm for some nonsmooth and nonconvex problems.
Paper Structure (30 sections, 8 theorems, 143 equations, 5 figures, 3 tables, 2 algorithms)

This paper contains 30 sections, 8 theorems, 143 equations, 5 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation and Related Work
  3. Contributions
  4. Notations
  5. Prliminaries
  6. Inexact Stochastic ADMM and Convergence Analysis
  7. The Update Rule of $\mathbf{y}$
  8. The Inexact Update of $\mathbf{x}$
  9. The Update Rule of $\boldsymbol{\lambda}$
  10. Convergence Analysis
  11. Global Convergence and Sublinear Convergence
  12. Linear Convergence Rate.
  13. Hybrid Stochastic Estimators for Inexact Update
  14. Inexact Update with Hybrid Stochastic Estimators
  15. Accelerated Hybrid Update for Inexact Subproblem
  16. ...and 15 more sections

Key Result

Lemma 1

Let $\left\{\mathbf{w}^k:= (\mathbf{x}^k,\mathbf{y}^k,\mathbf{\lambda}^k)\right\}$ be the iterate satisfying the conditions (y update criteria) and (x update criteria). Suppose Assumption assum 1 (a), (b), (c) hold and AL function is bounded below, we can choose the parameters in Algorithm alg 1 suc where $w>0$, $\hat{A}= 4\frac{1+\tau}{s\beta\sigma_{A}}\psi_1(s)c_x^2\beta^2,$$\sigma_{min}( \mathc

Figures (5)

  • Figure 1: Comparison of different hybrid parameter $\alpha$ for solving the nonconvex problem (\ref{['equ log with SCAD']}).
  • Figure 2: Test accuracy of \ref{['equ log with SCAD']} on ijcnn1 (left) and a9a (right).
  • Figure 3: The training loss of \ref{['equ log with SCAD']} on some real datasets.
  • Figure 4: Comparison of algorithms on training LeNet-5 on MNIST (upper) and CIFAR-10 (below).
  • Figure 5: The training loss of \ref{['equ fused Lasso']} on some real datasets.

Theorems & Definitions (23)

  • Definition 1: Clarke subgradient
  • Definition 2
  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Definition 3
  • Remark 1
  • Theorem 3
  • Corollary 1
  • Remark 2
  • ...and 13 more