Majorization-Minimization Dual Stagewise Algorithm for Generalized Lasso

TL;DR

This work proposes a majorization-minimization dual stagewise (MM-DUST) algorithm to efficiently trace out the full solution paths of the generalized lasso problem and analyzes the computational complexity of MM-DUST and establishes the uniform convergence of the approximated solution paths.

Abstract

The generalized lasso is a natural generalization of the celebrated lasso approach to handle structural regularization problems. Many important methods and applications fall into this framework, including fused lasso, clustered lasso, and constrained lasso. To elevate its effectiveness in large-scale problems, extensive research has been conducted on the computational strategies of generalized lasso. However, to our knowledge, most studies are under the linear setup, with limited advances in non-Gaussian and non-linear models. We propose a majorization-minimization dual stagewise (MM-DUST) algorithm to efficiently trace out the full solution paths of the generalized lasso problem. The majorization technique is incorporated to handle different convex loss functions through their quadratic majorizers. Utilizing the connection between primal and dual problems and the idea of ``slow-brewing'' from stagewise learning, the minimization step is carried out in the dual space through a sequence of simple coordinate-wise updates on the dual coefficients with a small step size. Consequently, selecting an appropriate step size enables a trade-off between statistical accuracy and computational efficiency. We analyze the computational complexity of MM-DUST and establish the uniform convergence of the approximated solution paths. Extensive simulation studies and applications with regularized logistic regression and Cox model demonstrate the effectiveness of the proposed approach.

Figures (9)

• Figure 1: Overview of the MM-DUST algorithm.
• Figure 2: Simulation: Solution paths of $\hbox{\boldmath $\beta$}$ and ${\bf u}$ with varying step sizes. The exact solution paths from glmnet are shown in red dashed lines, while the paths from MM-DUST are shown in black solid lines. The top figures are paths for the primal coefficient $\hbox{\boldmath $\beta$}$, while the bottom four are paths for the dual coefficient ${\bf u}$, with the x-axis as $\log(\lambda)$. The vertical dashed line marks the point when $\|\hat{{\bf u}}\|_{\infty}\leq \varepsilon$ and the algorithm stops.
• Figure 3: Simulation: Primal solution paths of $\hbox{\boldmath $\beta$}$ for varying step sizes. The settings are the same as in Figure \ref{['fig:path-logit']}. Note that (a) is further zoomed in on the y-axis to show the patterns clearer.
• Figure 4: Simulation: comparison with varied step sizes and the SPG algorithm with SNR=1. Reported are the measures over 100 repetitions.
• Figure 5: Simulation: full solution paths of $\hbox{\boldmath $\beta$}$ and ${\bf u}$ with different step sizes. The exact solution paths from glmnet are shown in red dashed lines, while the paths from MM-DUST are shown in black solid lines. The top four figures are the paths for the primal coefficient $\hbox{\boldmath $\beta$}$, with the x-axis as $\log(\lambda)$. The bottom four figures are the paths for the dual coefficient ${\bf u}$, with the x-axis as $\log(\lambda)$. The vertical dashed line marks the point when $\|\hat{{\bf u}}\|_{\infty}\leq \varepsilon$ and the algorithm stops.

Theorems & Definitions (5)

• Lemma 2.1
• Theorem 3.1
• Theorem 3.2
• Proposition 3.3
• Lemma D.1