Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An over-relaxed ADMM for separable convex programming and its applications to statistical learning

Renyuan Ni

TL;DR

An over-relaxed ADMM is developed integrated with a criterion to decide whether a relaxation step is implemented in the iteration, and the global convergence and $O(1/t)$ convergence rate of the over-relaxed ADMM are established.

Abstract

The alternating direction method of multipliers (ADMM) has been applied successfully in a broad spectrum of areas. Moreover, it was shown in the literature that ADMM is closely related to the Douglas-Rachford operator-splitting method, and can be viewed as a special case of the proximal point algorithm (PPA). As is well known, the relaxed version of PPA not only maintains the convergence properties of PPA in theory, but also numerically outperforms the original PPA. Therefore, it is interesting to ask whether ADMM can be accelerated by using the over-relaxation technique. In this paper, we answer this question affirmatively by developing an over-relaxed ADMM integrated with a criterion to decide whether a relaxation step is implemented in the iteration. The global convergence and $O(1/t)$ convergence rate of the over-relaxed ADMM are established. We also implement our proposed algorithm to solve Lasso and sparse inverse covariance selection problems, and compare its performance with the relaxed customized ADMM in \cite{CGHY} and the classical ADMM. The results show that our algorithm substantially outperforms the other two methods.

An over-relaxed ADMM for separable convex programming and its applications to statistical learning

TL;DR

An over-relaxed ADMM is developed integrated with a criterion to decide whether a relaxation step is implemented in the iteration, and the global convergence and convergence rate of the over-relaxed ADMM are established.

Abstract

The alternating direction method of multipliers (ADMM) has been applied successfully in a broad spectrum of areas. Moreover, it was shown in the literature that ADMM is closely related to the Douglas-Rachford operator-splitting method, and can be viewed as a special case of the proximal point algorithm (PPA). As is well known, the relaxed version of PPA not only maintains the convergence properties of PPA in theory, but also numerically outperforms the original PPA. Therefore, it is interesting to ask whether ADMM can be accelerated by using the over-relaxation technique. In this paper, we answer this question affirmatively by developing an over-relaxed ADMM integrated with a criterion to decide whether a relaxation step is implemented in the iteration. The global convergence and convergence rate of the over-relaxed ADMM are established. We also implement our proposed algorithm to solve Lasso and sparse inverse covariance selection problems, and compare its performance with the relaxed customized ADMM in \cite{CGHY} and the classical ADMM. The results show that our algorithm substantially outperforms the other two methods.
Paper Structure (10 sections, 6 theorems, 57 equations, 2 figures, 6 tables)

This paper contains 10 sections, 6 theorems, 57 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. The Over-relaxed ADMM
  3. Algorithm
  4. Variational characterization of \ref{['eq:1']}
  5. Preliminaries
  6. Convergence analysis
  7. Numerical Experiment
  8. Lasso Problem
  9. Sparse inverse covariance selection
  10. Conclusions

Key Result

lemma thmcounterlemma

For given $v^k=(y^k,\lambda^k)$, let $w^{k+1}$ be generated by the relaxed ADMM relax-ADMM and ${\tilde{w}}^k$ be defined by Tnotation. Then, we have

Figures (2)

  • Figure 1: The primal and dual residual for ADMM, relaxed ADMM and relaxed customized ADMM.
  • Figure 2: The primal and dual residual for ADMM, relaxed ADMM and relaxed customized ADMM. ($n=300$)

Theorems & Definitions (9)

  • lemma thmcounterlemma
  • lemma thmcounterlemma
  • proposition thmcounterproposition
  • proof
  • lemma thmcounterlemma
  • proof
  • lemma thmcounterlemma
  • theorem 1
  • proof