Bayes-Optimal Estimation in Generalized Linear Models via Spatial Coupling

TL;DR

An efficient approximate message passing (AMP) algorithm for estimation is proposed and it is proved that with a simple choice of spatially coupled design, the MSE of a carefully tuned AMP estimator approaches the asymptotic MMSE as the dimensions of the signal and the observation grow proportionally.

Abstract

We consider the problem of signal estimation in a generalized linear model (GLM). GLMs include many canonical problems in statistical estimation, such as linear regression, phase retrieval, and 1-bit compressed sensing. Recent work has precisely characterized the asymptotic minimum mean-squared error (MMSE) for GLMs with i.i.d. Gaussian sensing matrices. However, in many models there is a significant gap between the MMSE and the performance of the best known feasible estimators. In this work, we address this issue by considering GLMs defined via spatially coupled sensing matrices. We propose an efficient approximate message passing (AMP) algorithm for estimation and prove that with a simple choice of spatially coupled design, the MSE of a carefully tuned AMP estimator approaches the asymptotic MMSE in the high-dimensional limit. To prove the result, we first rigorously characterize the asymptotic performance of AMP for a GLM with a generic spatially coupled design. This characterization is in terms of a deterministic recursion (`state evolution') that depends on the parameters defining the spatial coupling. Then, using a simple spatially coupled design and a judicious choice of functions for the AMP algorithm, we analyze the fixed points of the resulting state evolution and show that it achieves the asymptotic MMSE. Numerical results for phase retrieval and rectified linear regression show that spatially coupled designs can yield substantially lower MSE than i.i.d. Gaussian designs at finite dimensions when used with AMP algorithms.

Figures (4)

• Figure 1: The entries of $\boldsymbol{A}$ are independent with $A_{ij} \sim \mathcal{N}(0,\frac{1}{m/\textsf{R}} W_{\textsf{r}(i), \textsf{c}(j)})$, where $\boldsymbol{W}$ is the base matrix. Here $\boldsymbol{W}$ is an $(\omega, \Lambda)$ base matrix with $\omega=3, \Lambda=7$ (see Definition \ref{['def:ome_lamb_rho']}). The white parts of $\boldsymbol{A}$ and $\boldsymbol{W}$ correspond to zeros.
• Figure 2: Potential function $U(x;\delta)$ for noiseless phase retrieval, for different values of sampling ratio $\delta$ and $x\in[0,\normalfont \textrm{Var}(X)]$. The signal prior is $P_X(a)=1- P_X(-a)=0.6$, with $a$ chosen so that $\normalfont \textrm{Var}(X)=1$. The red dots represent the largest stationary point and global minimizer of each function.
• Figure 3: MSE of Bayes-optimal GAMP for noiseless phase retrieval, with i.i.d. Gaussian and spatially coupled designs. The signal entries are drawn i.i.d. from $\{-a,a\}$ with probabilities $\{0.4,0.6\}$, and $a$ chosen such that the variance is $1$. Signal dimension $n=20000$, the base matrix parameters are $(\omega=6,\Lambda=40)$, and the empirical performance is obtained by averaging over 100 trials and the error bars represent $\pm1$ standard deviation.
• Figure 4: MSE of Bayes-optimal GAMP for noiseless rectified linear regression, with i.i.d. Gaussian and 2 different spatially coupled designs. The signal entries are drawn i.i.d. from $\{-b,0,b\}$ with probabilities $\{0.25,0.5,0.25\}$, and $b$ chosen such that the variance is $1$. Signal dimension $n=20000$ for i.i.d. GAMP. For SC-GAMP, the block dimension $n/\Lambda$ is fixed to $500$, and two sets of base matrix parameters are considered: $(\omega=6,\Lambda=40)$ and $(\omega=20,\Lambda=200)$. The empirical performance is obtained by averaging over 100 trials and the error bars represent $\pm1$ standard deviation.

Theorems & Definitions (15)

• Definition 2.1
• Theorem 1
• Definition 3.1: Potential function
• Theorem 2: MMSE for i.i.d. Gaussian design barbier2019optimal
• Theorem 3
• Corollary 3.1: Bayes-optimality of SC-GAMP
• proof
• Theorem 4
• Remark 5.1
• Corollary 5.1