Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region

TL;DR

The paper investigates gradient descent on matrix factorization under large learning rates, revealing a chaotic regime characterized by fractal convergence boundaries and initialization-sensitive outcomes near critical step sizes. It derives an exact critical step size for scalar factorization, shows that near criticality the converged minimizer can depend sensitively on initialization, and demonstrates that regularization induces fractal basin boundaries with a self-similar structure captured by a planar quotient dynamics. The results extend to general matrix factorization via invariant subspaces, where the dynamics decouple into scalar problems, and reveal that no simple implicit bias governs convergence near criticality. The work introduces a rigorous dynamical-systems framework for understanding the complexity of optimization in nonconvex, overparameterized models and suggests new directions for analyzing training dynamics beyond conventional stability notions.

Abstract

We examine gradient descent in matrix factorization and show that under large step sizes the parameter space develops a fractal structure. We derive the exact critical step size for convergence in scalar-vector factorization and show that near criticality the selected minimizer depends sensitively on the initialization. Moreover, we show that adding regularization amplifies this sensitivity, generating a fractal boundary between initializations that converge and those that diverge. The analysis extends to general matrix factorization with orthogonal initialization. Our findings reveal that near-critical step sizes induce a chaotic regime of gradient descent where the long-term dynamics are unpredictable and there are no simple implicit biases, such as towards balancedness, minimum norm, or flatness.

Figures (7)

• Figure 1: Left: Gradient descent applied to $L(u,v)=(u^\top v-1)^2+0.3(\|u\|_2^2+\|v\|_2^2)$ with $(u,v)\in \mathbb{R}^{10}$. Shown is a random two-dimensional slice of $\mathbb{R}^{10}$. Gray points are initializations from which the algorithm does not converge; other points are colored by the value of one of the coordinates of the converged minimizer. As we see, the convergence boundary is fractal, and the converged solution depends sensitively on the initialization when this is near the boundary. Right: Gradient descent applied to $L(x,y)=(xy-1)^2$ with $(x,y)\in \mathbb{R}^{2}$. The green star marks a balanced minimizer $p$, with its neighborhood $O_p$ depicted as a blue disk with black boundary. The blue region with dashed boundaries shows the preimage of $O_p$ under $\mathrm{GD}^6$. The green diamond marks an imbalanced minimizer, with its neighborhood and preimages shown in orange. For this problem the convergence boundary is smooth but the convergence point for initializations near the boundary is chaotic.
• Figure 2: Left: Gradient descent applied to $L(x,y)=(xy-1)^2$ with $(x,y)\in \mathbb{R}^{2}$. Blue lines and purple lines represent the basins of attraction of unstable minimizers and of the saddle $(\boldsymbol{0}, \boldsymbol{0})$, respectively. Right: For the same problem, we evenly sample initial values in an neighborhood on $\partial \mathcal{D}_\eta'$ (the blue square). We report the distributions of the squared norm and imbalance of the converged minimizer $(x^*,y^*)$, and of the number of iterations to reach a loss below $10^{-8}$.
• Figure 3: Gradient descent is applied to $L(u,v)=(uv-0.5)^2/2+0.1(u^2+v^2)$ where $(u,v)\in \mathbb{R}^2$. Left: the projected convergence boundary $T(\partial\mathcal{D}_\eta")$ is self-similar with degree three: it is covered by three smaller copies of itself (green, red, blue). Middle: The convergence boundary $\partial \mathcal{D}_\eta"$ consists of four replicates of $T(\partial\mathcal{D}_\eta")$, separated by gray lines. The only two minimizers are shown as red triangle and blue circle. Points are colored red if they converge to the red triangle, and blue if they converge to the blue circle. Right: the box-counting dimension of $T(\partial\mathcal{D}_\eta)$ is estimated as $1.249$.
• Figure 4: Gradient descent is applied to $L(u,v)=(uv-0.8)^2/2+\frac{\lambda}{2}(u^2+v^2)$, where $u,v\in \mathbb{R}$ and $\lambda\in \left\{0, 0.01, 0.1,0.5\right\}$. Blue points represent initializations that converge to a global minimizer; uncolored points represent initializations that do note converge. Red lines represent the basin of attraction of the saddle point $(0,0)$.
• Figure 5: Gradient descent is applied to $L(U,V)=\|U^\top V-Y\|_F^2/2$, where $U,V\in \mathbb{R}^{5 \times 4}$ and $Y$ is a diagonal matrix whose diagonal elements are randomly sampled from $[0,1]$. Four randomly sampled two-dimensional slices of the parameter space $\mathbb{R}^{40}$ are shown. The points are colored according to the squared Frobenius norm of the converged minimizer; uncolored points represent initializations that do not converge.

Theorems & Definitions (55)

• Theorem 1: Unregularized Scalar Factorization
• Proposition 2
• Theorem 3: Regularized Scalar Factorization
• Theorem 4
• Proposition 5
• Proposition 6
• proof
• Proposition 7
• proof
• Proposition 8: Proposition \ref{['prop:quotient']}