Adaptive Dimension Reduction for Overlapping Group Sparsity

TL;DR

The paper addresses dimension reduction for overlapping group sparsity in large-scale sparse optimization by introducing a lifted representation and two dual certificates (the LASSO certificate beta^* and the OGN certificate u^dagger). Based on these certificates, it proposes AdaDROPS, an adaptive dimension reduction scheme that can be plugged into Primal-Dual splitting, ADMM, and variable projection, with convergence guarantees. The authors provide extensive theoretical analysis and empirical validation on standard datasets (e.g., LIBSVM and wavelet-based imaging), demonstrating significant speedups, including cases with more than an order of magnitude acceleration. This work extends safe screening methods to the overlapping-group setting and offers a practical framework for accelerating first-order solvers in complex structured sparsity problems, with potential extensions to other nonsmooth models.

Abstract

Typical dimension reduction techniques for nonoverlapping sparse optimization involve screening or sieving strategies based on a dual certificate derived from the first-order optimality condition, approximating the gradients or exploiting certain inherent low-dimensional structure of the sparse solution. In comparison, dimension reduction rules for overlapping group sparsity are generally less developed because the subgradient structure is more complex, making the link between sparsity pattern and the dual variable indirect due to the non-separability. In this work, we propose new dual certificates for overlapping group sparsity and a novel adaptive scheme for identifying the support of the overlapping group LASSO. We demonstrate how this scheme can be integrated into and significantly accelerate existing algorithms, including Primal-Dual splitting method, alternating direction method of multipliers and a recently developed variable projection scheme based on over-parameterization. We provide convergence analysis of the method and verify its practical effectiveness through experiments on standard datasets.

Figures (6)

• Figure 1: Illustration of overlapping group norm: i) $x\in\mathbb{R}^4$ has $1$ nonzero element, the grouping is $\mathcal{G} = \{ \{1,2\}, \{2,3\}, \{3,4\} \}$; ii) $z\in\mathbb{R}^6$ has nonoverlapping grouping $\mathcal{J} = \{ \{1,2\}, \{3,4\}, \{5,6\} \}$; iii) The active groups of $x$ is $\mathcal{I}_x = \{1,2\}$; iv) The extended supports are $\mathcal{E}_x = \{1,2\},\mathcal{E}_z = \{1,2,3,4\}$ and $\mathcal{E}_L = \{1,2,3\}$.
• Figure 2: Illustration of $\widehat{L}$. Consider $\mathcal{G} = \{ \{1,2\}, \{2,3\}, \{3,4\}\}$ with $x_{G_1} \neq 0, x_{G_2}=x_{G_3}=0$. For any $u\in \partial {||} Lx {||}_{1,2}$, we have $u_2=0$. The extended support $\mathcal{E}_z = \{1,2\}, \mathcal{E}_L = \{1\}$. The subtraction in $\widehat{L}$ is removing the 2nd row (corresponding to $x_2$, which overlaps between zero and nonzero groups).
• Figure 3: Comparison of overlapping group LASSO over three datasets. First row: runtime; Second row: size of linear system.
• Figure 4: Runtime comparison on two synthetic images. First row: $n=128$; Second row: $n=256$.
• Figure 5: Comparison of multi-task LASSO on different datasets.

Theorems & Definitions (32)

• Remark 3.1
• Definition 3.2: Effective lifting operator
• Proposition 3.3
• Definition 3.4: LASSO certificate
• Definition 3.5: OGN certificate
• Remark 3.6: Nonoverlapping case
• Proposition 3.7
• Remark 3.8: Nondegeneracy and tightness
• Remark 3.9
• Proposition 3.10: OGN certificate