Sharpness of Minima in Deep Matrix Factorization: Exact Expressions

TL;DR

The paper derives an exact closed-form expression for the maximum Hessian eigenvalue (sharpness) at any minimizer in overparameterized deep matrix factorization, resolving an open question and clarifying how sharpness governs gradient-based dynamics. It shows that flat minima in depth-2 matrices are characterized by spectral-norm balancing rather than Frobenius-norm balancing and provides a simplified expression for depth-2 and scalar-factorization cases. The authors connect the exact sharpness to an empirically observed escape phenomenon during gradient descent and discuss the limitations of various sharpness measures, arguing for a robust, universal notion. Overall, the work deepens theoretical understanding of loss-landscape geometry in deep linear networks and informs interpretations of optimization dynamics and generalization in overparameterized settings.

Abstract

Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eigenvalue of the Hessian of the loss, which measures the sharpness of the landscape. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in general overparameterized deep matrix factorization (i.e., deep linear neural network training) problems, resolving an open question posed by Mulayoff & Michaeli (2020). This expression uncovers a fundamental property of the loss landscape of depth-2 matrix factorization problems: a minimum is flat if and only if it is spectral-norm balanced, which implies that flat minima are not necessarily Frobenius-norm balanced. Furthermore, to complement our theory, we empirically investigate an escape phenomenon observed during gradient-based training near a minimum that crucially relies on our exact expression of the sharpness.

Figures (7)

• Figure 1: GD dynamics with different step sizes, $\eta \geq 2/\lambda_{\max}(\nabla^2 \mathcal{L}(\bm{w}^*))$, indicated by different colors, are initialized within a radius of $10^{-15}$ from the minimum in the direction of the Hessian eigenvector corresponding to the largest eigenvalue, for depth-$2$ matrix factorization, $\bm{M} = \bm{L}\bm{R}^\top$, of a random Gaussian matrix, where $\bm{L} \in \mathbb{R}^{10 \times 20}$ and $\bm{R} \in \mathbb{R}^{20 \times 20}$. The vector $\bm{v}_1$ denotes the eigenvector of the Hessian corresponding to the largest eigenvalue, while $\bm{v}_N$ denotes the eigenvector corresponding to the smallest eigenvalue. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\bm{w}^*))$ is computed using the closed-form expression derived in Corollary \ref{['remark:twolayer']}.
• Figure 2: GD dynamics with different step sizes indicated by different colors are initialized within a radius of $10^{-12}$ from the minimum in the direction of the Hessian eigenvector corresponding to the largest eigenvalue, for a 15-layer overparameterized scalar factorization of a random scalar. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\bm{w}^*))$ is computed using the closed-form expression derived in Theorem \ref{['theorem:warmupdeepoverparameterizedscalarfactorization']}.
• Figure 3: Contour map of the loss landscape around a minimum in a 15-layer overparameterized scalar factorization of a random scalar. GD with different step sizes $\eta$, indicated by different colors, is initialized within a radius of $10^{-9}$ from the minimum, in the direction of the Hessian eigenvector corresponding to the largest eigenvalue. The vector $\bm{v}_1$ denotes the eigenvector of the Hessian corresponding to the largest eigenvalue, while $\bm{v}_N$ denotes the eigenvector corresponding to the smallest eigenvalue. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\bm{w}^*))$ is computed using the closed-form expression derived in Theorem \ref{['theorem:warmupdeepoverparameterizedscalarfactorization']}.
• Figure 4: GD dynamics with different step sizes indicated by different colors for general matrix factorization, $\bm{M} = \bm{L}\bm{R}^\top$, of a random Gaussian matrix, where $\bm{L} \in \mathbb{R}^{10 \times 30}$ and $\bm{R} \in \mathbb{R}^{20 \times 30}$. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\bm{w}^*))$ is computed using the closed-form expression derived in Corollary \ref{['remark:twolayer']}.
• Figure 5: GD dynamics with different step sizes indicated by different colors for general matrix factorization, $\bm{M} = \bm{L}\bm{R}^\top$, of a random Gaussian matrix, where $\bm{L} \in \mathbb{R}^{25 \times 30}$ and $\bm{R} \in \mathbb{R}^{20 \times 30}$. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\bm{w}^*))$ is computed using the closed-form expression derived in Corollary \ref{['remark:twolayer']}.

Theorems & Definitions (18)

• Definition 1
• Lemma 2
• Lemma 3
• Theorem 4
• Theorem 5
• Corollary 6
• proof
• Corollary 7
• proof
• Corollary 8