On Partly Smoothness, Activity Identification and Faster Algorithms of $L_1$ over $L_2$ Minimization

TL;DR

This paper establishes the partly smooth property of L_1 over L_2$ minimization relative to an active manifold ${\cal M}$ and demonstrates its prox-regularity property and proposes a novel heuristic algorithm framework that combines ADMM (or ADMM) with a globalized semismooth Newton method tailored for the active manifold.

Abstract

The $L_1/L_2$ norm ratio arose as a sparseness measure and attracted a considerable amount of attention due to three merits: (i) sharper approximations of $L_0$ compared to the $L_1$; (ii) parameter-free and scale-invariant; (iii) more attractive than $L_1$ under highly-coherent matrices. In this paper, we first establish the partly smooth property of $L_1$ over $L_2$ minimization relative to an active manifold ${\cal M}$ and also demonstrate its prox-regularity property. Second, we reveal that ADMM$_p$ (or ADMM$^+_p$) can identify the active manifold within a finite iterations. This discovery contributes to a deeper understanding of the optimization landscape associated with $L_1$ over $L_2$ minimization. Third, we propose a novel heuristic algorithm framework that combines ADMM$_p$ (or ADMM$^+_p$) with a globalized semismooth Newton method tailored for the active manifold ${\cal M}$. This hybrid approach leverages the strengths of both methods to enhance convergence. Finally, through extensive numerical simulations, we showcase the superiority of our heuristic algorithm over existing state-of-the-art methods for sparse recovery.

Figures (5)

• Figure 1: Comparison between ADMM$_p$ and HAFAM$_T$ ($T=5,10,20,30$) for Gaussian matrices with $r=0.8$ (top row) and O-DCT matrices with $F=10$ (bottom row). The evolution of RErr and KKT$_R$ from HAFAM$_T$ consistently exhibits a linear reduction when the active manifold is identified, highlighting its accelerated convergence over ADMM$_p$.
• Figure 2: The performance profiles of ADMM$_p$ and HAFAM$_T$ ($T=5,10,20,30$) for $90$ different problems. HAFAM$_{30}$ excels in KKT$_R$ performance among these comparing algorithms, while HAFAM$_{5}$ takes the least CPU time.
• Figure 3: The evolution of the RErr from ADMM$_p$ and HAFAM$_T$ ($T = 5,\;30$) with varying $\tau$ values under a noise level of $\sigma = 0.05$ (left) and the $\text{IAcc}({\mathbf{x}}^{I},{{{\mathbf{x}}^*}})$ values for different $\tau$ settings under two noise levels (right). Specifically, the values of $\text{IAcc}({\mathbf{x}}^{I},{{{\mathbf{x}}^*}})$ for $\tau = 0,\sigma,2\sigma,3\sigma$ are provided: (a) [$\sigma = 0.01, T = 5$]: 0.83, 0.87, 0.89, 0.91; (b) [$\sigma = 0.01, T = 30$]: 0.94, 0.95, 0.95, 0.96; (c) [$\sigma= 0.05, T = 5$]: 0.76, 0.88, 0.92, 0.94; (d) [$\sigma = 0.05, T = 30$]: 0.92, 0.96, 0.97, 0.97.
• Figure 4: Each plot depicts the average of ${\text{IAcc}}({\hat{\mathbf{x}}},{\mathbf{x}}_T^{I})$ across 50 random instances for 256 different problems, where $n=1024$, $m=16:16:256$, and $s=1:1:16$. Top row: $T=5$ (left) and $T=10$ (right); Bottom row: $T=20$ (left) and $T=30$ (right). For $T=5,10,20,30$, the minimum value of ${\text{IAcc}}({\hat{\mathbf{x}}},{\mathbf{x}}_T^{I})$ in each plot are $0.98,0.98,0.98,0.98$; and the maximum value of ${\text{IAcc}}({\hat{\mathbf{x}}},{\mathbf{x}}_T^{I})$ in each plot are $1.00,1.00,1.00,1.00$.
• Figure 5: Comparison among $L_1/L_2$ via ADMM$_p$ and HAFAM$_T$, $L_1$ and $L_{1/2}$. The evolution of test MSE (TMSE) and relative error (RelErr) on diabetes dataset: ratio = $8:2$ (left); ratio = $7:3$ (right). ADMM$_p$ and HAFAM$_T$ outperform $L_1$ and $L_{1/2}$ in terms of TMSE and RelErr. While ADMM$_p$ and HAFAM$_T$ initially overlap in TMSE and RelErr, HAFAM$_T$ quickly converges to a smaller value when the active manifold is identified.

Theorems & Definitions (27)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Lemma 1
• proof
• Remark 1
• Proposition 1
• proof
• Remark 2