Precise Asymptotics for Spectral Methods in Mixed Generalized Linear Models

TL;DR

This work analyzes the precise asymptotics of spectral methods for recovering two independent signals in a mixed generalized linear model with Gaussian design. In the proportional regime, it derives a master theorem that characterizes the joint distribution of the linear estimator, spectral estimators, and the signals, enabling exact overlaps and optimal combinations. The authors design optimal preprocessing for both linear and spectral estimators, establish universal spectral-threshold bounds, and show that a Bayes-optimal linear-spectral combination can substantially outperform individual estimators in relevant regimes. Their approach combines random matrix theory, free probability, and generalized AMP state evolution, and is validated by numerical experiments on mixed linear regression and phase retrieval. The results illuminate when spectral methods are advantageous, how to tune them, and how to merge them with simple linear estimates for best-in-class performance in high dimensions.

Abstract

In a mixed generalized linear model, the goal is to learn multiple signals from unlabeled observations: each sample comes from exactly one signal, but it is not known which one. We consider the prototypical problem of estimating two statistically independent signals in a mixed generalized linear model with Gaussian covariates. Spectral methods are a popular class of estimators which output the top two eigenvectors of a suitable data-dependent matrix. However, despite the wide applicability, their design is still obtained via heuristic considerations, and the number of samples $n$ needed to guarantee recovery is super-linear in the signal dimension $d$. In this paper, we develop exact asymptotics on spectral methods in the challenging proportional regime in which $n, d$ grow large and their ratio converges to a finite constant. This allows us optimize the design of the spectral method, and combine it with a simple linear estimator, to minimize the estimation error. Our characterization exploits a mix of tools from random matrices, free probability and the theory of approximate message passing algorithms. Numerical simulations for mixed linear regression and phase retrieval demonstrate the advantage enabled by our analysis over existing designs of spectral methods.

Figures (6)

• Figure 1: Noiseless mixed linear regression with mixing parameter (i.e., probability that a sample corresponds to $x_1^*$) $\alpha = 0.6$. Overlaps with the first signal $x_1^*$ (left) and the second signal $x_2^*$ (right), computed via simulation ("sim.") and the theoretical prediction ("pred."), are plotted as a function of the aspect ratio $\delta =n/d$. The signal dimension is $d=2000$. We note that our optimal spectral estimator enables weak recovery of both signals at a smaller $\delta$ than existing spectral estimators designed for non-mixed data. E.g., in the right panel, our optimal spectral estimator starts weakly recovering $x_2^*$ when $\delta > 3.1$ while other spectral estimators require at least $\delta > 5.3$.
• Figure 1: Plot of $\psi(\lambda;\Delta),\varphi(\lambda),\zeta(\lambda;\Delta)$ as functions of $\lambda$ with $\Delta\in\{\delta,\delta_1,\delta_2\}$.
• Figure 1: Spectral estimators for noiseless mixed linear regression, with mixing parameter $\alpha \in \{ 0.6, 0.8\}$. Optimal spectral estimators given by \ref{['eqn:rk-spec-preproc-same']} are used. Overlaps with both signals $x_1^*,x_2^*$, computed from simulation ("sim.") and prediction ("pred."), are plotted as a function of the aspect ratio $\delta$. Same numerics apply to noiseless phase retrieval (see \ref{['rk:lin-regr-phase-retrieval-coincide']}).
• Figure 2: Smallest $\delta$ required by different spectral estimators to weakly recover signals for noiseless mixed phase retrieval. The spectral threshold is plotted as a function of a varying mixing parameter $\alpha \in [0.6, 0.8]$. Our optimal spectral estimator always attains the lowest threshold. We note that these thresholds remain the same for noiseless mixed linear regression, due to the design of the corresponding estimators.
• Figure 2: Spectral estimators for mixed linear regression and mixed phase retrieval. Optimal spectral estimators (\ref{['eqn:opt-prec-spec-lin-regr-main', 'eqn:opt-prec-spec-phase-retr-main']}) are used. Overlaps with the first signal $x_1^*$ (left plot) and with both signals $x_1^*,x_2^*$ (right plot), computed from simulation ("sim.") and prediction ("pred."), are plotted as a function of the aspect ratio $\delta$.

Theorems & Definitions (45)

• Theorem 3.1: Master theorem on joint distribution
• Remark 3.2: Equivalence to convergence of empirical distribution
• Remark 3.3: What if either the linear or spectral estimator is ineffective
• Remark 3.4: $v_i(D)$ estimates $x_i^*$
• Remark 3.5: Sign calibration of spectral estimator
• Remark 3.6: Master theorem for $\alpha = 1/2$
• Corollary 3.7: Bayes-optimal linear-spectral combination
• Corollary 3.8: Overlaps, linear
• Proof 1
• Remark 3.9: Overlap of linear estimator does not approach $1$