The phase diagram of compressed sensing with $\ell_0$-norm regularization

TL;DR

This work tackles noiseless compressed sensing with $l_0$-norm regularization to improve recovery at high compression under Gaussian design. It develops two ASP-based algorithms, ASP and ASP_o, whose behavior is exactly trackable by State Evolution in the large-system limit, yielding a precise phase diagram for perfect recovery and outperforming LASSO. Through replica analysis, the paper reveals replica-symmetric and 1RSB minima, explaining easy vs hard recovery phases and showing the 1RSB fixed points are stable at low regularization. These results quantify recovery limits, provide practical unsupervised algorithms for high-undersampling regimes, and point to future work on noisy measurements and broader inference problems.

Abstract

Noiseless compressive sensing is a two-steps setting that allows for undersampling a sparse signal and then reconstructing it without loss of information. The LASSO algorithm, based on $\lone$ regularization, provides an efficient and robust to address this problem, but it fails in the regime of very high compression rate. Here we present two algorithms based on $\lzero$-norm regularization instead that outperform the LASSO in terms of compression rate in the Gaussian design setting for measurement matrix. These algorithms are based on the Approximate Survey Propagation, an algorithmic family within the Approximate Message Passing class. In the large system limit, they can be rigorously tracked through State Evolution equations and it is possible to exactly predict the range compression rates for which perfect signal reconstruction is possible. We also provide a statistical physics analysis of the $\lzero$-norm noiseless compressive sensing model. We show the existence of both a replica symmetric state and a 1-step replica symmmetry broken (1RSB) state for sufficiently low $\lzero$-norm regularization. The recovery limits of our algorithms are linked to the behavior of the 1RSB solution.

Figures (4)

• Figure 1: Plots showing that AMP and PGD algorithms convergence near the true signal in the unrealistic setting of starting very close to it (fully-informed initialization). We simulated the AMP algorithm (triangles), and AMP followed by PGD iterations (circles) with learning rate $\delta=0.6$. In all instances, the system size is $N=5\times 10^3$ and the signal density is $\rho_o=0.6$. (Top left) The rescaled loss function $\left\langle\mathcal{L}_\lambda(\bm{x}) \right\rangle /{\lambda N}=e/\lambda+\rho$ as a function of $\lambda$. (Top right) Density $\rho$ of the output configuration. (Bottom) Reconstruction error, i.e. the $l_2$-norm distance between the signal and the output configurations. As expected the agreement between the two configurations improves when $\lambda$ is lowered.
• Figure 2: (Left) Overlap at convergence with the true signal when running the ASP (orange), ${\rm ASP_o}$ (blue) and LASSO (red) algorithms at different values of the regularization prefactor $\lambda$. The dashed lines correspond to finite size simulations while the full lines correspond to the infinite size prediction from State Evolution (SE). For this experiment, we set $\rho_o=0.6$ and $\alpha=0.87$. The ${\rm ASP_o}$ parameter is fixed to $\xi=0.7$. (Right) The quantity $V$ (for ${\rm ASP_o}$ and LASSO) and $V_1+s V_0$ (ASP) in the same experimental setting.
• Figure 3: (Left) Phase diagram for perfect recovery in compressed sensing. The $\ell_1$ and Bayesian Optimal lines give the algorithmic thresholds for $\ell_1$ based reconstruction and Bayesian optimal AMP respectively. Above those lines, the corresponding algorithms succeed with high probability for large systems. The $\ell_0$ (ASP, ${\rm ASP_o}$) line instead, gives the recovery threshold for our algorithms as predicted by state evolution. Lastly, the IT line is the information theoretic threshold for perfect recovery. It also corresponds to the ultimate limit for $\ell_0$-based recovery algorithms. In the region between the IT and the $\ell_0$ (ASP, ${\rm ASP_o}$) line though, it is algorithmically hard to find the global optimum of the $\ell_0$-based loss. (Right) We plot the rescaled cost $\left\langle\mathcal{L}_\lambda(\bm{x})\right\rangle /{\lambda N}=e/\lambda+\rho$ as a function of $\lambda$, for $\rho_o=0.6$ and several values of $\alpha$. We show the predictions from the informed AMP state evolution (i.e. $m^{t=0}\approx q^{t=0}\approx\rho_o$) and the uninformed ASP one. In the regime for which the signal recovery is easy ($\alpha>0.83$) both saddle points describe the true signal as $\lambda$ goes to zero, we thus have $e/\lambda+\rho\to \rho_o$. At lower $\alpha$ instead, the uninformed saddle point no longer describes the true signal, but configurations at larger density.
• Figure 4: Stability condition (type I as defined in Antenucci2019) for the 1RSB saddle point both in the regimes where it yields signal recovery ($\alpha=0.9$) and where it does not ($\alpha=0.7$). Regardless of the regime, we observe that the 1RSB saddle point is stable when $\lambda$ is low enough, i.e. the curves lay above zero.