Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

Abstract

Standard optimization theories struggle to explain grokking, where generalization occurs long after training convergence. While geometric studies attribute this to slow drift, they often overlook the interaction between the optimizer's noise structure and landscape curvature. This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a ``Spectral Gating'' mechanism that regulates the transition from memorization to generalization. We find that AdamW operates as a variance-gated stochastic system. Grokking is constrained by a stability condition: the generalizing solution resides in a sharp basin ($λ_{max}^H$) initially inaccessible under low-variance regimes. The ``delayed'' phase represents the accumulation of gradient variance required to lift the effective stability ceiling, permitting entry into this sharp manifold. Our ablation studies identify three complexity regimes: (1) \textbf{Capacity Collapse} ($P < 23$), where rank-deficiency prevents structural learning; (2) \textbf{The Variance-Limited Regime} ($P \approx 41$), where generalization waits for the spectral gate to open; and (3) \textbf{Stability Override} ($P > 67$), where memorization becomes dimensionally unstable. Furthermore, we challenge the "Flat Minima" hypothesis for algorithmic tasks, showing that isotropic noise injection fails to induce grokking. Generalization requires the \textit{anisotropic rectification} unique to adaptive optimizers, which directs noise into the tangent space of the solution manifold.

Figures (14)

• Figure 1: Stability-Complexity Phase Diagram. We define three regimes based on the time-to-generalization: (1) Capacity Collapse ($P < 23$), where rank-deficiency prevents the representation of the solution; (2) Variance-Limited Regime ($29 \le P \le 59$), where generalization is delayed by spectral gating; and (3) Stability Override ($P \ge 67$), where the high dimensionality of the memorization manifold forces immediate structural learning. The white dashed line marks $\epsilon = \sigma_{noise}$, representing the intrinsic gradient noise level.
• Figure 2: Capacity Collapse ($P=17$). Despite aggressive stability tuning ($\epsilon \to 10^{-15}$), the optimizer fails to locate the generalizing minimum. The dynamics exhibit stochastic tunneling without convergence, indicating that the embedding space lacks the geometric capacity to separate the task's Fourier modes.
• Figure 3: Non-Monotonic Generalization Dynamics. Generalization time peaks at intermediate complexity ($P=41$, Red). Hard tasks ($P=97$, Teal) generalize immediately (Stability Override), while easy tasks ($P=23$, Purple) stagnate. The results reveal a "Complexity Valley" where the competition between the entropic pull of memorization and the spectral gate of generalization is maximized.
• Figure 4: Radial Stationarity. (a) AdamW generates a constant diffusive "push" (peach line) that counters the restorative force. (b) The restorative drift stabilizes, confirming a fixed radial orbit. (c) Stronger weight decay (peach) forces a lower-norm equilibrium, compressing the search space onto the manifold musat2025geometry.
• Figure 5: Acceleration via Constraint. (a) Higher weight decay (Peach) significantly accelerates the phase transition. (b) Strong regularization suppresses the peak gradient variance, enforcing a disciplined search in the tangent space.