Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Solvers for Sparse Subspace Clustering

Farhad Pourkamali-Anaraki, James Folberth, Stephen Becker

TL;DR

This paper tackles scalable sparse subspace clustering (SSC) by unifying SSC-$\ell_1$ and SSC-$\ell_0$ under a proximal gradient framework that supports affine constraints. It derives efficient proximity operators for all four variants (with/without affine constraints for both $\ell_1$ and $\ell_0$), achieving $\mathcal{O}(n^2)$ time and memory for the full matrix case and favorable $\mathcal{O}(pn^2)$ operations with matrix-inversion-lambda-inspired tricks. Convergence guarantees are provided for both convex SSC-$\ell_1$ and nonconvex SSC-$\ell_0$, and extensive experiments show robustness to parameter choices and superior scalability relative to ADMM and OMP, especially on large-scale or affine-subspace problems. The proposed methods substantially reduce computational cost for large datasets while maintaining accurate subspace-preserving representations, enabling practical SSC in real-world, high-dimensional settings.

Abstract

Sparse subspace clustering (SSC) clusters $n$ points that lie near a union of low-dimensional subspaces. The SSC model expresses each point as a linear or affine combination of the other points, using either $\ell_1$ or $\ell_0$ regularization. Using $\ell_1$ regularization results in a convex problem but requires $O(n^2)$ storage, and is typically solved by the alternating direction method of multipliers which takes $O(n^3)$ flops. The $\ell_0$ model is non-convex but only needs memory linear in $n$, and is solved via orthogonal matching pursuit and cannot handle the case of affine subspaces. This paper shows that a proximal gradient framework can solve SSC, covering both $\ell_1$ and $\ell_0$ models, and both linear and affine constraints. For both $\ell_1$ and $\ell_0$, algorithms to compute the proximity operator in the presence of affine constraints have not been presented in the SSC literature, so we derive an exact and efficient algorithm that solves the $\ell_1$ case with just $O(n^2)$ flops. In the $\ell_0$ case, our algorithm retains the low-memory overhead, and is the first algorithm to solve the SSC-$\ell_0$ model with affine constraints. Experiments show our algorithms do not rely on sensitive regularization parameters, and they are less sensitive to sparsity misspecification and high noise.

Efficient Solvers for Sparse Subspace Clustering

TL;DR

This paper tackles scalable sparse subspace clustering (SSC) by unifying SSC- and SSC- under a proximal gradient framework that supports affine constraints. It derives efficient proximity operators for all four variants (with/without affine constraints for both and ), achieving time and memory for the full matrix case and favorable operations with matrix-inversion-lambda-inspired tricks. Convergence guarantees are provided for both convex SSC- and nonconvex SSC-, and extensive experiments show robustness to parameter choices and superior scalability relative to ADMM and OMP, especially on large-scale or affine-subspace problems. The proposed methods substantially reduce computational cost for large datasets while maintaining accurate subspace-preserving representations, enabling practical SSC in real-world, high-dimensional settings.

Abstract

Sparse subspace clustering (SSC) clusters points that lie near a union of low-dimensional subspaces. The SSC model expresses each point as a linear or affine combination of the other points, using either or regularization. Using regularization results in a convex problem but requires storage, and is typically solved by the alternating direction method of multipliers which takes flops. The model is non-convex but only needs memory linear in , and is solved via orthogonal matching pursuit and cannot handle the case of affine subspaces. This paper shows that a proximal gradient framework can solve SSC, covering both and models, and both linear and affine constraints. For both and , algorithms to compute the proximity operator in the presence of affine constraints have not been presented in the SSC literature, so we derive an exact and efficient algorithm that solves the case with just flops. In the case, our algorithm retains the low-memory overhead, and is the first algorithm to solve the SSC- model with affine constraints. Experiments show our algorithms do not rely on sensitive regularization parameters, and they are less sensitive to sparsity misspecification and high noise.

Paper Structure

This paper contains 21 sections, 3 theorems, 28 equations, 6 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Notation
  3. Review of Sparse Subspace Clustering
  4. The Proposed Methods
  5. Proximal Gradient Descent Framework
  6. Algorithms for SSC-$\ell_1$ and SSC-$\ell_0$
  7. Proximity Operators for Each Case
  8. $\ell_1$ proximity operator
  9. $\ell_1$ proximity operator with affine constraint
  10. $\ell_0$ projection
  11. $\ell_0$ projection with affine constraint
  12. Convergence Results
  13. SSC-$\ell_1$
  14. SSC-$\ell_0$
  15. Numerical Experiments
  16. ...and 6 more sections

Key Result

Proposition 4

The problem ex:prox can be solved exactly in $\mathcal{O}( n \log n)$ flops.

Figures (6)

  • Figure 1: SSC-$\ell_1$ on the Extended Yale B data set for (a) $K=2$ and (b) $K=3$ clusters. For each case, three metrics are used from left to right: value of the objective function, subspace preserving error, and clustering error. The legends in (a) and (b) are the same.
  • Figure 2: Clustering error as a function of $\rho$.
  • Figure 3: Clustering error of SSC-$\ell_1$ on synthetic data.
  • Figure 4: Running time (logarithmic scale) of SSC-$\ell_1$ on synthetic data for varying $n$.
  • Figure 5: Clustering error of SSC-$\ell_0$ on synthetic data for varying $\sigma$ (noise level).
  • ...and 1 more figures

Theorems & Definitions (10)

  • Remark 1
  • Remark 2
  • Remark 3
  • Proposition 4
  • proof
  • Theorem 5
  • Remark 6
  • Remark 7
  • Theorem 8
  • Remark 9