Local minima of the empirical risk in high dimension: General theorems and convex examples

TL;DR

This work develops a unified Kac-Rice framework to analyze high-dimensional ERM with Gaussian covariates in the proportional regime, deriving sharp bounds on the number and location of local minima via a variational rate function ${\Phi}_{\mathrm{gen}}$. By specializing to convex losses with ridge regularization, the authors obtain rate-trivialization results and a clear variational procedure that yields unique minimizers and precise asymptotics for the empirical risk and Hessian spectrum; they further apply the framework to exponential families and multinomial regression, with explicit fixed-point equations for the optimal limiting measures. The theory is validated through synthetic data and Fashion-MNIST experiments using random feature mappings, showing strong agreement with the predicted estimation, training, and Hessian-spectrum behavior. The results bridge high-dimensional statistics, random matrix theory, and non-convex ERM analysis, and set the stage for sharp non-convex results in a companion paper. Overall, the paper provides a principled probabilistic and variational lens to understand the landscape of high-dimensional ERM and the spectral properties of the Hessian at minimizers, with implications for estimation accuracy and optimization tractability in large-scale models.

Abstract

We consider a general model for high-dimensional empirical risk minimization whereby the data $\mathbf{x}_i$ are $d$-dimensional Gaussian vectors, the model is parametrized by $\mathbfΘ\in\mathbb{R}^{d\times k}$, and the loss depends on the data via the projection $\mathbfΘ^\mathsf{T}\mathbf{x}_i$. This setting covers as special cases classical statistics methods (e.g. multinomial regression and other generalized linear models), but also two-layer fully connected neural networks with $k$ hidden neurons. We use the Kac-Rice formula from Gaussian process theory to derive a bound on the expected number of local minima of this empirical risk, under the proportional asymptotics in which $n,d\to\infty$, with $n\asymp d$. Via Markov's inequality, this bound allows to determine the positions of these minimizers (with exponential deviation bounds) and hence derive sharp asymptotics on the estimation and prediction error. As a special case, we apply our characterization to convex losses. We show that our approach is tight and allows to prove previously conjectured results. In addition, we characterize the spectrum of the Hessian at the minimizer. A companion paper applies our general result to non-convex examples.

Figures (8)

• Figure 1: Train/test error (log loss), estimation error, and classification error of ridge regularized multinomial regression, for $(k+1)=3$ symmetric classes, as a function of the regularization parameter $\lambda$ for several values of $\alpha$. Empirical results are averaged over 100 independent trials, with $d = 250$. Continuous lines are theoretical predictions obtained by solving numerically the system \ref{['eq:FP_multinomial']}.
• Figure 2: Train/test error (log loss), estimation error, and classification error of unregularized multinomial regression, for $(k+1)=3$ symmetric classes, as a function of $\alpha$ for different values of ${\boldsymbol R}_{00}$ specified in the text. Empirical results are averaged over 100 independent trials, with $d = 250$.
• Figure 3: Histograms of the empirical spectral distribution of the Hessian at the MLE for multinomial regression with three symmetric classes, in $d=250$ dimensions, aggregated over $100$ independent realizations. From left to right, $\alpha = 3$, $\alpha = 5$, $\alpha=10$, and $\alpha = 20$. Blue lines represent the theoretical distribution derived from Proposition \ref{['prop:multinomial']}.
• Figure 4: Performance of multinomial regression on the Fashion-MNIST dataset for $(k+1)=3$ classes, as a function of $\alpha$. We construct feature vectors ${\boldsymbol x}_i$ using a random one-layer neural network, as discussed in Section \ref{['sec:Fashion-MNIST']}.
• Figure 5: Histograms: Empirical spectral distribution of the Hessian at the MLE. Here we use use Fashion-MNIST data for $(k+1)=3$ classes, and construct feature vectors ${\boldsymbol x}_i$ using a random one-layer neural network, as discussed in Section \ref{['sec:Fashion-MNIST']}.

Theorems & Definitions (105)

• Example 1: Multinomial regression
• Example 2: Two-layer neural networks
• Remark 2.1
• Theorem 1
• Remark 2.2
• Remark 2.3
• Remark 2.4: Universality
• Remark 2.5: Tightness
• Remark 2.6: Ridge regularization
• Remark 2.7: Case $\alpha\le 1$