Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving Sparsity Constrained PCA, Regression, and QCQP via the Spartrahedron

Diego Cifuentes, Zhuorui Li

Abstract

Sparsity is a fundamental modeling principle in statistics, signal processing, and data science. However, optimization with sparsity constraints is notoriously difficult. We introduce a new convex relaxation framework for {sparse quadratically constrained quadratic programs} (QCQPs), a class that subsumes sparse regression, sparse principal component analysis (PCA), and related problems. Our approach is based on a novel convex cone, the spartrahedron, which exactly characterizes sparsity at the matrix level. This leads to a semidefinite programming (SDP) relaxation that is tight whenever its solution is rank-one, providing a simple certificate of global optimality. We establish theoretical guarantees, including approximation bounds and exactness regions for sparse PCA and sparse ridge regression, as well as a general stability result under perturbations. Numerical experiments on sparse PCA, sparse regression, RIP constant estimation, and sparse canonical correlation analysis (CCA) demonstrate the practical success of our methods.

Solving Sparsity Constrained PCA, Regression, and QCQP via the Spartrahedron

Abstract

Sparsity is a fundamental modeling principle in statistics, signal processing, and data science. However, optimization with sparsity constraints is notoriously difficult. We introduce a new convex relaxation framework for {sparse quadratically constrained quadratic programs} (QCQPs), a class that subsumes sparse regression, sparse principal component analysis (PCA), and related problems. Our approach is based on a novel convex cone, the spartrahedron, which exactly characterizes sparsity at the matrix level. This leads to a semidefinite programming (SDP) relaxation that is tight whenever its solution is rank-one, providing a simple certificate of global optimality. We establish theoretical guarantees, including approximation bounds and exactness regions for sparse PCA and sparse ridge regression, as well as a general stability result under perturbations. Numerical experiments on sparse PCA, sparse regression, RIP constant estimation, and sparse canonical correlation analysis (CCA) demonstrate the practical success of our methods.
Paper Structure (34 sections, 33 theorems, 183 equations, 10 figures)

This paper contains 34 sections, 33 theorems, 183 equations, 10 figures.

Table of Contents

  1. Introduction
  2. The spartrahedron and other convex relaxations
  3. Comparison of the cones
  4. Extreme rays and dual cones
  5. Relaxations of sparse QCQPs
  6. Local stability analysis
  7. Conditions guaranteeing local stability
  8. Sufficient conditions for exactness
  9. Sparse PCA
  10. Relaxation gap
  11. Exact recovery
  12. Wigner and Wishart spiked models
  13. Application to Restricted Isometry Property
  14. Sparse Ridge Regression
  15. Relaxation gap
  16. ...and 19 more sections

Key Result

Lemma 2.1

For a vector $x\in \mathbb{R}^n$, then $x x^T \in \mathcal{S}_{n,k}$ if and only if $\|x\|_0\leq k$. In particular, $\mathop{\mathrm{conv}}\nolimits(Q_{n,k}) \subseteq \mathcal{S}_{n,k}$

Figures (10)

  • Figure 1: Comparison of $\mathcal{S}_{3,2}$, $\mathcal{S}_{3,2}^{\ell_1}$ and $\mathcal{S}_{n,k}^{\textit{\tiny{Z}}}$
  • Figure 2: Wigner spiked model: $n=50,\lambda=12$
  • Figure 3: Wishart spiked model: $n = 50, N=2000, \beta=1.5$
  • Figure 4: Comparison of upper and lower bound with Pitprops dataset ($n = 13$).
  • Figure 5: Comparison of upper and lower bound with Eisen-1 dataset ($n = 79$).
  • ...and 5 more figures

Theorems & Definitions (75)

  • Lemma 2.1
  • proof
  • Theorem 2.1
  • proof
  • Lemma 2.2
  • proof
  • Lemma 2.3
  • proof
  • Example 2.2: $n=3,k=2$
  • Theorem 2.3
  • ...and 65 more