Anderson Accelerated Operator Splitting Methods for Convex-nonconvex Regularized Problems

TL;DR

This paper addresses the computation of convex-nonconvex (CNC) regularized problems by casting CNC-regularized least squares as monotone inclusions and applying four operator-splitting methods (DRS, FBS, FBFS, DYS). It develops an Anderson-accelerated framework (A2OS) with regularization and safeguarding to guarantee global convergence while delivering substantial practical speedups on sparse regression, matrix regression/completion, and sparse group lasso tasks. Theoretical results establish convergence under standard monotone-operator assumptions, and extensive experiments demonstrate significant reductions in iterations and runtime without sacrificing accuracy. The methods extend CNC applicability to broader domains, offering a scalable, robust toolkit for convexity-preserving nonconvex regularization in signal processing and statistical learning.

Abstract

Convex-nonconvex (CNC) regularization is a novel paradigm that employs a nonconvex penalty function while maintaining the convexity of the entire objective function. It has been successfully applied to problems in signal processing, statistics, and machine learning. Despite its wide application, the computation of CNC regularized problems remains challenging and under-investigated. To fill the gap, we study several operator splitting methods and their Anderson accelerated counterparts for solving least squares problems with CNC regularization. We establish the global convergence of the proposed algorithm to an optimal point and demonstrate its practical speed-ups in various applications.

Figures (6)

• Figure 1: The residual norm trajectories of DRS, FBS, and FBFS in solving the GMC (top row) and the group GMC (bottom row) problems when $\lambda=0.1\lambda_{\max}$. Wall clock run times in seconds in parentheses next to method names in the legend boxes.
• Figure 2: A binary coefficient matrix in a cross shape (left panel) and a checkerboard coefficient matrix generated by the checkerboard function in Matlab (right panel).
• Figure 3: Maximum step size of FBS and FBFS when $\lVert A \rVert_2=1$.
• Figure 4: $R^2$ on the validation set as $\lambda/\lambda_{\max}$ varies for sparse group lasso.
• Figure 5: The residual norm trajectories of DYS using the original FPI, naive AA, and regularized and safeguarded AA (Algorithm 1) at $\lambda = 10^{-0.2} \lambda_{\max}$.

Theorems & Definitions (14)

• Remark 1
• Remark 2
• Remark 3
• Proposition 2.1
• Remark 4
• Remark 5
• Theorem 3.1
• proof
• Lemma 1
• proof