Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares

Christian Kümmerle, Johannes Maly

TL;DR

This work tackles the problem of recovering matrices that are simultaneously row-sparse and low-rank from linear measurements. It introduces a non-convex iteratively reweighted least squares (IRLS) method with a multi-structure weight operator that jointly promotes low rank and sparsity, and proves local quadratic convergence under an $(r,s)$-restricted isometry property with near-optimal sample complexity $m=\Omega(r(s+n_2))$. The method is given a variational majorize-minimize interpretation, providing a rigorous link between the iterative updates and a smoothed objective, and it is shown to identify ground-truth structures efficiently in practice. Numerically, IRLS outperforms state-of-the-art approaches (SPF, RiemAdaIHT) in data-efficiency across Gaussian, rank-one, and Fourier-type measurements, and exhibits robust performance under moderate noise with favorable convergence behavior and a self-balancing between the two structure-promoting terms.

Abstract

We propose a new algorithm for the problem of recovering data that adheres to multiple, heterogeneous low-dimensional structures from linear observations. Focusing on data matrices that are simultaneously row-sparse and low-rank, we propose and analyze an iteratively reweighted least squares (IRLS) algorithm that is able to leverage both structures. In particular, it optimizes a combination of non-convex surrogates for row-sparsity and rank, a balancing of which is built into the algorithm. We prove locally quadratic convergence of the iterates to a simultaneously structured data matrix in a regime of minimal sample complexity (up to constants and a logarithmic factor), which is known to be impossible for a combination of convex surrogates. In experiments, we show that the IRLS method exhibits favorable empirical convergence, identifying simultaneously row-sparse and low-rank matrices from fewer measurements than state-of-the-art methods. Code is available at https://github.com/ckuemmerle/simirls.

Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares

TL;DR

This work tackles the problem of recovering matrices that are simultaneously row-sparse and low-rank from linear measurements. It introduces a non-convex iteratively reweighted least squares (IRLS) method with a multi-structure weight operator that jointly promotes low rank and sparsity, and proves local quadratic convergence under an -restricted isometry property with near-optimal sample complexity . The method is given a variational majorize-minimize interpretation, providing a rigorous link between the iterative updates and a smoothed objective, and it is shown to identify ground-truth structures efficiently in practice. Numerically, IRLS outperforms state-of-the-art approaches (SPF, RiemAdaIHT) in data-efficiency across Gaussian, rank-one, and Fourier-type measurements, and exhibits robust performance under moderate noise with favorable convergence behavior and a self-balancing between the two structure-promoting terms.

Abstract

We propose a new algorithm for the problem of recovering data that adheres to multiple, heterogeneous low-dimensional structures from linear observations. Focusing on data matrices that are simultaneously row-sparse and low-rank, we propose and analyze an iteratively reweighted least squares (IRLS) algorithm that is able to leverage both structures. In particular, it optimizes a combination of non-convex surrogates for row-sparsity and rank, a balancing of which is built into the algorithm. We prove locally quadratic convergence of the iterates to a simultaneously structured data matrix in a regime of minimal sample complexity (up to constants and a logarithmic factor), which is known to be impossible for a combination of convex surrogates. In experiments, we show that the IRLS method exhibits favorable empirical convergence, identifying simultaneously row-sparse and low-rank matrices from fewer measurements than state-of-the-art methods. Code is available at https://github.com/ckuemmerle/simirls.
Paper Structure (33 sections, 13 theorems, 134 equations, 7 figures, 1 algorithm)

This paper contains 33 sections, 13 theorems, 134 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contribution
  3. Related Work
  4. Sparse and Low-Rank Recovery
  5. Iteratively Reweighted Least Squares
  6. Notation
  7. IRLS for Sparse and Low-Rank Reconstruction
  8. How to Combine Sparse and Low-Rank Weighting
  9. Local Quadratic Convergence of IRLS
  10. IRLS as Quadratic Majorize-Minimize Algorithm
  11. Discussion of Computational Complexity
  12. Numerical Evaluation
  13. Performance in Low-Measurement Regime
  14. Sensitivity to Parameter Choice
  15. Convergence Behavior
  16. ...and 18 more sections

Key Result

Theorem 2.5

\newlabelthm:QuadraticConvergence0 Let $\mathbf{X}_\star \in \mathcal{M}_{r,s}$ be a fixed ground-truth matrix that is $s$-row-sparse and of rank $r$. Let linear observations $\mathbf{y} = \mathcal{A}(\mathbf{X}_\star)$ be given and assume that $\mathcal{A}$ has the $(r,s)$-RIP with $\delta \in (0 where $c_{\left\| \mathcal{A} \right\|_{2\to 2}} = \sqrt{1 + \tfrac{\left\| \mathcal{A} \right\|_{2\

Figures (7)

  • Figure 1: Left column: RiemAdaIHT, center: SPF, right: IRLS. Phase transition experiments with $n_1=256$, $n_2=40$, $r=1$, Gaussian measurements. Algorithmic hyperparameters informed by model order knowledge (i.e., $\widetilde{r}=r$ and $\widetilde{s}=s$ for IRLS). White corresponds to empirical success rate of $1$, black to $0$.
  • Figure 1: Left column: RiemAdaIHT, center: SPF, right: IRLS. Success rates for the recovery of low-rank and row-sparse matrices from random rank-one measurements. First row: Rank-$1$ ground truth $\mathbf{X}_\star$ (cf. \ref{['fig:Performance_Rank1']}. Second row: Rank-$5$ ground truth $\mathbf{X}_\star$ (cf. \ref{['fig:Performance_Rank5']}).
  • Figure 2: Left column: RiemAdaIHT, center: SPF, right: IRLS. First row: As in \ref{['fig:Performance_Rank1']}, but for data matrix $\mathbf{X}_\star$ of rank $r=5$. Second row: As first row, but hyper-parameters $r$ and $s$ are overestimated as $\widetilde{r}= 2r = 10$, $\widetilde{s} = \lfloor 1.5s \rfloor$.
  • Figure 2: Left column: RiemAdaIHT, center: SPF, right: IRLS. Success rates for the recovery of low-rank and row-sparse matrices from Fourier rank-one measurements. First row: Rank-$1$ ground truth $\mathbf{X}_\star$. Second row: Rank-$5$ ground truth $\mathbf{X}_\star$ (cf. \ref{['fig:Performance_Rank5']}).
  • Figure 3: Comparison of convergence rate. Setting as in \ref{['fig:Performance_Rank5']} with $s=40$ and $m=1125$.
  • ...and 2 more figures

Theorems & Definitions (25)

  • Definition 2.1
  • Remark 2.2
  • Remark 2.3
  • Definition 2.4
  • Theorem 2.5: Local Quadratic Convergence
  • Theorem 2.6
  • Lemma B.1
  • Lemma B.2
  • Proof 1
  • Lemma B.3
  • ...and 15 more