A Unified Approach to Statistical Estimation Under Nonlinear Observations: Tensor Estimation and Matrix Factorization

TL;DR

This work introduces a unified RAIC-based framework for statistical estimation from nonlinear observations $y_i=f_i(\langle\mathbf{a}_i,\boldsymbol{x}\rangle)$. By constructing a gradient from data and proving restricted approximate invertibility, the authors establish convergence guarantees for projected gradient descent, Riemannian gradient descent on low-Tucker-rank tensors, and factorized gradient descent for low-rank matrices, under Gaussian designs. The framework yields minimax-optimal or near-optimal statistical rates across a spectrum of models, including tensor single index models, tensor generalized linear models (e.g., tensor logistic regression), tensor phase retrieval (noisy and one-bit variants), and one-bit tensor sensing, with novel tensor-estimation results and efficient algorithms. Comprehensive numerical studies, including a hyperspectral data experiment, validate the theory and illustrate practical applicability. Overall, RAIC serves as a nonlinear RIP-like condition that unifies algorithmic convergence and statistical optimality across vectors, matrices, and tensors in nonlinear observation regimes.

Abstract

We consider the estimation of some parameter $\mathbf{x}$ living in a cone from the nonlinear observations of the form $\{y_i=f_i(\langle\mathbf{a}_i,\mathbf{x}\rangle)\}_{i=1}^m$. We develop a unified approach that first constructs a gradient from the data and then establishes the restricted approximate invertibility condition (RAIC), a condition that quantifies how well the gradient aligns with the ideal descent step. We show that RAIC yields linear convergence guarantees for the standard projected gradient descent algorithm, a Riemannian gradient descent algorithm for low Tucker-rank tensor estimation, and a factorized gradient descent algorithm for asymmetric low-rank matrix estimation. Under Gaussian designs, we establish sharp RAIC for the canonical statistical estimation problems of single index models, generalized linear models, noisy phase retrieval, and one-bit compressed sensing. Combining the convergence guarantees and the RAIC, we obtain a set of optimal statistical estimation results, including, to our knowledge, the first minimax-optimal and computationally efficient algorithms for tensor single index models, tensor logistic regression, (local) noisy tensor phase retrieval, and one-bit tensor sensing. Moreover, several other results are new or match the best known guarantees. We also provide simulations and a real-data experiment to illustrate the theoretical results.

Figures (11)

• Figure 1: Geometric illustration of the RAIC.
• Figure 2: Accurate tensor estimation in single index models and one-bit sensing. We run $20$ iterations in each independent trial. Figures \ref{['fig:tsimtrun']} and \ref{['fig:tsimsigmod']} show that increasing sample size and decreasing $(n,r)$ lead to more accurate estimation, consistent with the rate $O(\sqrt{T_{df}/m})$. In one-bit tensor sensing, the log-log plot in Figure \ref{['fig:tsim1b']} confirms that the specialized Algorithm \ref{['alg:t1bcs']} achieves the faster rate $O(T_{df}/m)$ compared with Algorithm \ref{['alg:tsim']}.
• Figure 3: Accurate tensor estimation in logistic regression, probit regression and poisson regression. We run 50 iterations. The results suggest that the normalization step is not actually necessary for logistic regression and probit regression (Figures \ref{['fig:logi']}, \ref{['fig:probit']}). Figure \ref{['fig:poisson']} indicates that the algorithm performs well for Poisson regression even though its theoretical analysis lies beyond the scope of our current framework.
• Figure 4: Tensor phase retrieval with initialization $\mathbfcal{X}_0=\mathcal{P}_{\mathbb{S}_{\rm F}}(H^\mathbf{n}_\mathbf{r}(\rho \mathbfcal{X} + (1-\rho){\rm Unif}(\mathbb{S}_{\rm F})))$. We test noiseless case with $m=3000$ (otherwise stated) under a number of $\rho$'s in Figure \ref{['fig:noiseless']}: we find that $\rho\ge0.17$ leads to local contraction toward the desired ground truth, whereas $\rho\le 0.16$ becomes trapped at some local stationary point distant from $\mathbfcal{X}$; specifically, random initialization fails, in contrast to the low-dimensional phase retrieval chen2019gradient and tensor regression via RGN luo2023low. In Figures \ref{['fig:noisy']}--\ref{['fig:noisy2']} we test noisy tensor phase retrieval with i.i.d. $\mathcal{N}(0,\sigma^2)$ noise: Figure \ref{['fig:noisy']} shows that the algorithm converges in only a few ($<5$) iterations; Figure \ref{['fig:noisy2']} verifies that increasing $m$ and decreasing $(n,r,\sigma)$ both lead to smaller estimation errors, consistent with the Frobenius norm rate $\tilde{O}(\sigma\sqrt{T_{df}/m})$ in Theorem \ref{['thm:TPR']}. Interestingly, comparing the blue and orange curves in Figures \ref{['fig:noiseless']} and \ref{['fig:noisy']} finds that larger $m$ also yields faster convergence rate, consistent with the linear contraction rate $\tilde{O}(\sqrt{T_{df}/m})$ in Theorem \ref{['thm:TPR']}.
• Figure 5: Accurate matrix estimation in nonlinear models via factorization. The results in Figure \ref{['fig:mf1']} for noiseless matrix phase retrieval $\{y_i=|\langle\mathbf{A}_i,\mathbf{X}\rangle|\}_{i=1}^m,~\mathbf{X}\in M^{n,n}_r$ confirm that FGD is capable of exactly finding the underlying matrix; they also show that a good initialization is crucial and the rebalancing step is not necessary in practice. In Figure \ref{['fig:mf2']}, we test noisy matrix phase retrieval $\{y_i=|\langle\mathbf{A}_i,\mathbf{X}\rangle|+\epsilon_i\}_{i=1}^m$ with $\epsilon_i\sim\mathcal{N}(0,\sigma^2\mathbf{I}_n)$; the results therein are consistent with the error rate $\tilde{O}(\sigma\sqrt{rn/m})$; see Supplement \ref{['app:fgdpr']}. One-bit matrix sensing $\{y_i=\mathop{\mathrm{sign}}\nolimits(\langle\mathbf{A}_i,\mathbf{X}\rangle)\}_{i=1}^m$ is considered Figure \ref{['fig:mf3']}. We run FGD with the gradient in single index model or the more specialized gradient for one-bit sensing, showing that they achieve the rates $O(m^{-1/2})$ and $O(m^{-1})$, respectively.

Theorems & Definitions (88)

• Proposition 2.1: See, e.g., rauhut2017low
• Definition 3.1
• Theorem 3.1: Convergence of PGD
• Remark 3.1
• Theorem 3.2: Convergence of RGD
• Remark 3.2
• Theorem 3.3: Convergence of FGD
• Theorem 4.1: Tensor regression
• Remark 4.1
• Theorem 5.1: RAIC for Single Index Models