An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Models

TL;DR

This work analyzes why training trajectories of diverse networks concentrate on a low-dimensional training-manifold in prediction space. It introduces a tractable linear-model framework and shows the PCA covariance $P(T)$ decomposes as $P(T) = P1_sigma_w(T) + P1_y(T) - P2(T)$, with a unit-rank $P2(T)$, and demonstrates that the spectrum is governed by the input-spectrum slope $c$, the initialization ratio sigma_star/sigma_w, and the number of gradient steps $T$. It derives explicit eigenvalue bounds (e.g., lambda^{P2} = O(1/T^2) and a closed-form for lambda_i^{sigma_w}) and phase boundaries predicting hyper-ribbon dimensionality, also extending to SGD, ridge, and kernel methods. The results illuminate how data sloppiness and training dynamics constrain prediction-space exploration, offering practical guidelines for hyper-parameter choices to manage model complexity and generalization.

Abstract

Recent experiments have shown that training trajectories of multiple deep neural networks with different architectures, optimization algorithms, hyper-parameter settings, and regularization methods evolve on a remarkably low-dimensional "hyper-ribbon-like" manifold in the space of probability distributions. Inspired by the similarities in the training trajectories of deep networks and linear networks, we analytically characterize this phenomenon for the latter. We show, using tools in dynamical systems theory, that the geometry of this low-dimensional manifold is controlled by (i) the decay rate of the eigenvalues of the input correlation matrix of the training data, (ii) the relative scale of the ground-truth output to the weights at the beginning of training, and (iii) the number of steps of gradient descent. By analytically computing and bounding the contributions of these quantities, we characterize phase boundaries of the region where hyper-ribbons are to be expected. We also extend our analysis to kernel machines and linear models that are trained with stochastic gradient descent.

Figures (8)

• Figure 1: Left: The manifold of models along training trajectories of networks with different configurations (architectures denoted by different colors, optimization algorithms, hyper-parameters, and regularization mechanisms) is effectively low-dimensional for CIFAR-10. This is a partial reproduction using the data from mao2024training. Linear networks are trained upon images directly (dark green), after pre-processing using one layer ("Scattering-1" in lighter green) and two layers ("Scattering-2" in the lightest green) of a scattering transform mallat2012group. With typical weight initializations all models begin training near $P_0$, where every sample is assigned equal probability to belong to every class (marked by hand to guide the reader). They progress to the truth $P_*$ (not seen here) to different degrees. All nonlinear deep networks in this experiment achieve zero training error. While linear networks do not fit the data perfectly, the manifolds swept by linear networks are quite similar to those of deep networks. These common low-dimensional manifolds in probability space are the inspiration of this paper. Right: A quantitative analysis of the inPCA embedding in terms of the explained stress which characterizes how well pairwise distances between points are preserved after the embedding. When the inPCA embedding is computed using all the points on the left, the explained stress of the first two dimensions is about 71% (red), about the same when inPCA is computed using only nonlinear models (orange). We can compute inPCA using points corresponding to only linear models and embed all other models (green) or only the nonlinear models (blue) into this space. While there are clearly differences between the manifolds of linear and nonlinear models (blue curve is lower), it is remarkable that nonlinear models can be faithfully represented in the embedding constructed using linear models (green line is close to red and orange). Our analysis in this paper that focuses on linear models is therefore a meaningful insight into the manifolds of nonlinear models.
• Figure 2: Real-world data are sloppy Eigenvalues of the empirical correlation matrices of feature representations of different types of data exhibit a sharp drop-off among the first few eigenvectors. This is characteristic of multi-parameter models fit to data, but here it arises in real-world data. For CIFAR-10 krizhevsky2009learning, we use raw pixel values as the features. For WikiText-103 merity2016pointer, the text is broken into token sequences of length 16, and BERT devlin2019bert embeddings from the last hidden layer are concatenated to form a feature vector. For sounds corresponding to the Bird category in AudioSet gemmeke2017audio, each 1-second audio segment is treated as a sample, with spectrograms serving as feature vectors.
• Figure 3: Eigenvalues of PCA of points on training trajectories of linear regression and eigenvalues of different important contributions to it. There is a rapid decrease in the PCA eigenvalues (black), and thus these points admit a low-dimensional representation with high explained-variance in the first few dimensions, similar to linear and nonlinear models in \ref{['fig:panel']}. The PCA covariance matrix $P(T)$ after $T$ gradient descent steps can be decomposed analytically into two important contributions: $P^y(T)$ that depends upon the regression targets and $P^{\sigma_w}(T)$ that depends upon the distribution of initial weights. The sharp initial decay in the eigenvalues of $P(T)$ is well-approximated by $P^y$ while the decay in the tail is controlled by $P^{\sigma_w}$. Details in \ref{['fig:decomposition']}.
• Figure 4: Contributions to the eigenspectrum of the PCA matrix coming from weight initialization computed using the bound in \ref{['eq:w1_eval_bound']} (dotted) and numerical computation of the corresponding term in \ref{['eq:P']} (bold) with $\sigma_w^2 = 1$, $\alpha = 1$ and $c = 0.5$. For very large training times $T \gg \lambda_1^K / \lambda_n^K$, the eigenvalue $\lambda_i^{\sigma_w}$ corresponds to the minimum in the numerator being 1 for any $i$. This means that $\lambda_i^{\sigma_w}$ decays to $\sim \sigma_w^2/2\alpha$ as the index $i$ increases, at a rate determined by the decay of $\lambda_i^K$. From \ref{['eq:w1_eval_bound']}, for $T \ll \lambda_1^K / \lambda_n^K$ and $i < \ln(2T\alpha) / c$, the minimum in the numerator is 1, and we again have a similar decay. For larger values of $i$, the minimum comes from the other term and therefore $\lambda_i^{\sigma_w}$ decays to much smaller values $\sim \frac{1}{(2/\lambda_n^K - \alpha)}$. This is the lower envelope of the curves above. The limit $T \to 0$ corresponds to the eigenvalues of $\sigma_w^2 K$ and therefore reflects the sloppy decay in the input correlation matrix.
• Figure 5: The tail of the eigenspectrum of $P_1(T)$ is well-approximated by the contribution coming from the weight initializations $P_1^{\sigma_w}(T)$, while the head is well approximated by the contribution coming from the targets $P_1^y(T)$. These eigenvalues were computed for $d=100$ dimensional data with slope $c=0.2$ for the eigenvalues of the input correlation matrix, after fitting $n = 50$ samples for $T = 50$ iterations with initialization variance $\sigma_w^2 = 0.1$ and variance of the ground-truth weights being $\sigma_*^2 = 2$, this experiment uses $N=100$ random initializations.

Theorems & Definitions (3)

• Lemma 1
• Lemma 2
• Lemma 3