Sparsity-Constraint Optimization via Splicing Iteration

TL;DR

This work tackles sparsity-constrained optimization by introducing SCOPE, a tuning-free iterative method that leverages splicing between active and inactive coordinates in low-dimensional subspaces. By ensuring monotone decrease of the objective and exploiting a relevance-based swapping mechanism, SCOPE achieves linear convergence and exact support recovery under mild restricted convexity/smoothness assumptions, both with and without RIP-type conditions. Theoretical guarantees are complemented by applications to compressed sensing, sparse classification, and sparse Ising models, plus empirical evidence of substantial speedups over exact solvers and competing methods. An open-source implementation, skscope, demonstrates practical scalability and competitive performance in large-scale settings.

Abstract

Sparsity-constraint optimization has wide applicability in signal processing, statistics, and machine learning. Existing fast algorithms must burdensomely tune parameters, such as the step size or the implementation of precise stop criteria, which may be challenging to determine in practice. To address this issue, we develop an algorithm named Sparsity-Constraint Optimization via sPlicing itEration (SCOPE) to optimize nonlinear differential objective functions with strong convexity and smoothness in low dimensional subspaces. Algorithmically, the SCOPE algorithm converges effectively without tuning parameters. Theoretically, SCOPE has a linear convergence rate and converges to a solution that recovers the true support set when it correctly specifies the sparsity. We also develop parallel theoretical results without restricted-isometry-property-type conditions. We apply SCOPE's versatility and power to solve sparse quadratic optimization, learn sparse classifiers, and recover sparse Markov networks for binary variables. The numerical results on these specific tasks reveal that SCOPE perfectly identifies the true support set with a 10--1000 speedup over the standard exact solver, confirming SCOPE's algorithmic and theoretical merits. Our open-source Python package skscope based on C++ implementation is publicly available on GitHub, reaching a ten-fold speedup on the competing convex relaxation methods implemented by the cvxpy library.

Figures (2)

• Figure 1: The boxplot of accuracy (Left panel) and runtime (Right panel). Upper panel: $n$ increases when $p$ and $s$ is fixed at $p=100$ and $s=10$; Middle panel: $p$ increases when fixing $n=50$, $s=10$; Bottom panel: increasing $s$ but fixing $n=50$, $p=100$. Note that the runtime of the two methods is limited to 1000 seconds. The experiment was independently repeated 100 times.
• Figure 2: The accuracy (upper panel) and runtime (bottom panel) as the sample size increases from 100 with step size 100. All experiments were independently repeated 100 times. The $y$-axis for runtime is $\log_{10}$-transformed.

Theorems & Definitions (34)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Definition 1: Restricted Strong Convexity and Restricted Strong Smoothness
• Theorem 1
• proof
• Lemma 1
• Lemma 2
• Lemma 3