Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning

TL;DR

The paper investigates the implicit regularization of gradient descent in matrix factorization, challenging norm-based characterizations. It proves that gradient flow with infinitesimal initialization is generically equivalent to a greedy low-rank learning procedure (GLRL) for depth-2 models and extends the analysis to deeper factorizations, where depth enhances the likelihood of rank minimization despite practical initialization scales. A key contribution is a concrete counterexample to the nuclear-norm conjecture, demonstrating that GD can favor low-rank solutions not captured by nuclear norm minimization alone. The framework links end-to-end gradient dynamics to a phased, rank-increase algorithm and provides both theoretical and empirical support that depth yields multi-phase low-rank growth, offering a more expressive picture of implicit regularization beyond norm-based descriptions. These insights pave the way for broader applications of GLRL in understanding optimization biases and for exploring extensions to deep neural networks.

Abstract

Matrix factorization is a simple and natural test-bed to investigate the implicit regularization of gradient descent. Gunasekar et al. (2017) conjectured that Gradient Flow with infinitesimal initialization converges to the solution that minimizes the nuclear norm, but a series of recent papers argued that the language of norm minimization is not sufficient to give a full characterization for the implicit regularization. In this work, we provide theoretical and empirical evidence that for depth-2 matrix factorization, gradient flow with infinitesimal initialization is mathematically equivalent to a simple heuristic rank minimization algorithm, Greedy Low-Rank Learning, under some reasonable assumptions. This generalizes the rank minimization view from previous works to a much broader setting and enables us to construct counter-examples to refute the conjecture from Gunasekar et al. (2017). We also extend the results to the case where depth $\ge 3$, and we show that the benefit of being deeper is that the above convergence has a much weaker dependence over initialization magnitude so that this rank minimization is more likely to take effect for initialization with practical scale.

Figures (9)

• Figure 1: The trajectory of depth-$2$ GD, ${\bm{W}}_\text{GD}(t)$, converges to the trajectory of GLRL, ${\bm{W}}_\text{GLRL}(t)$, as the initialization scale goes to $0$. We plot $\mathrm{dist}(t) = \min_{t' \in \mathcal{T}} \| {\bm{W}}_\text{GD}(t) - {\bm{W}}_\text{GLRL}(t') \|_{\mathrm{F}}$ for different initialization scale $\| {\bm{W}}(0) \|_{\mathrm{F}}$, where $\mathcal{T}$ is a discrete subset of $\mathbb{R}$ that $\delta$-covers the entire trajectory of GLRL: $\max_t\min_{t'\in \mathcal{T}}\left\| {\bm{W}}_\text{GLRL}(t) - {\bm{W}}_\text{GLRL}(t') \right\|_{\mathrm{F}} \leq \delta$ for $\delta \approx 0.00042$. For each $\| {\bm{W}}(0) \|_{\mathrm{F}}$, we run $20$ random seeds and plot them separately. The ground truth ${\bm{W}}^* \in \mathbb{R}^{20 \times 20}$ is a randomly generated rank-$3$ matrix with $\| {\bm{W}}^* \|_{\mathrm{F}} = 20$. $30\%$ entries are observed. See more in \ref{['sec:exp-setup']}.
• Figure 2: GD passes by the same set of critical points as GLRL when the initialization scale is small, and gets much closer to the critical points when $L\ge 3$. Depth-$2$ GD requires a much smaller initialization scale to maintain small low-rankness. Here the ground truth matrix ${\bm{W}}^* \in \mathbb{R}^{20 \times 20}$ is of rank $3$ as stated in \ref{['sec:exp-setup']}. In this case, GLRL has $3$ phases and $4$ critical points $\{\overline{{}{\bm{W}}}_r\}_{r=0}^3$, where $\overline{{}{\bm{W}}}_0 = {\bm{0}}$ and $\overline{{}{\bm{W}}}_3 = {\bm{W}}^*$. For each depth $L$ and initialization scale $\| {\bm{W}}(0) \|_{\mathrm{F}}$, we plot the distance between the current step of GD and the closest critical point of GLRL, $\min_{0 \le r \le 3}\| {\bm{W}}_{\text{GD}}(t)- \overline{{}{\bm{W}}}_r \|_{\mathrm{F}}$, the norm of full gradient, $\| \nabla_{{\bm{U}}_{1:L}} \mathcal{L}({\bm{U}}_{1:L}) \|_{\mathrm{F}}$ and the $(r+1)$-low-rankness of ${\bm{W}}_{\text{GD}}(t)$ with $r:= \mathop{\mathrm{arg\,min}}\limits_{0\le r \le 3}\| {\bm{W}}_{\text{GD}}(t)- \overline{{}{\bm{W}}}_r \|_{\mathrm{F}}$.
• Figure 3: GD with small initialization outperforms R1MP and minimal nuclear norm solution on synthetic data with low-rank ground truth. Solid (dotted) curves correspond to test (training) loss. Here the loss $f({\bm{W}}):=\frac{1}{d^2} \| {\bm{W}} - {\bm{W}}^* \|_{\mathrm{F}}^2$ and $f({\bm{0}}) = 1$. We run 10 random seeds for GD and plot them separately (most of them overlap).
• Figure 4: Using ${\epsilon} {\bm{v}}_1{\bm{v}}_1^\top$ (denoted by "rank $1$") as initialization makes GD much closer to GLRL compared to using random initialization (denoted by "rank $d$"), where ${\bm{v}}_1$ is the top eigenvector of $-\nabla f({\bm{0}})$. We take a fixed reference matrix on the trajectory of GLRL with constant norm and plot the distance of GD with each initialization to it respectively..
• Figure 5: Deep matrix factorization encourages GF to find low rank solutions at a much practical initialization scale, e.g. $10^{-3}$. Here the ground truth is rank-$3$. For each setting, we run 5 different random seeds. The solid curves are the mean and the shaded area indicates one standard deviation. We observe that performance of GD is quite robust to its initialization. Note that for $L > 2$, the shaded area with initialization scale $10^{-7}$ is large, as the sudden decrement of loss occurs at quite different continuous times for different random seeds in this case.

Theorems & Definitions (102)

• Conjecture 1.1: gunasekar2017implicit, informal
• Definition 5.1: Trajectory of GLRL
• Definition 5.2
• Theorem 5.3
• proof : Proof sketch
• Lemma 5.4
• Theorem 5.6
• Theorem 5.8
• Example 5.9: Counter-example of \ref{['conj:nuclear']}, gunasekar2017implicit
• Theorem 5.10