The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon

TL;DR

The paper investigates the grokking generalization paradox by uncovering the Slingshot Mechanism, an empirical, late-stage cyclic dynamic in adaptive optimizers that causes abrupt norm shifts in the last classification layer. Through extensive experiments across FCNs, Transformers, ViTs, and synthetic setups, it shows Slingshots correlate with grokking and occur broadly with Adam, AdamW, and RMSProp, but are suppressed by higher epsilon values or alternative optimizers. The work reveals that norm growth and curvature spikes drive phase transitions, suggesting a surprising inductive bias of adaptive optimizers that challenges existing convergence theories and motivates new norm-control strategies. Overall, Slingshot dynamics appear to be a general, optimizer-dependent mechanism that can enhance generalization, but also introduce instability and longer training times, underscoring the need for theoretical grounding and practical stabilization methods.

Abstract

The grokking phenomenon as reported by Power et al. ( arXiv:2201.02177 ) refers to a regime where a long period of overfitting is followed by a seemingly sudden transition to perfect generalization. In this paper, we attempt to reveal the underpinnings of Grokking via a series of empirical studies. Specifically, we uncover an optimization anomaly plaguing adaptive optimizers at extremely late stages of training, referred to as the Slingshot Mechanism. A prominent artifact of the Slingshot Mechanism can be measured by the cyclic phase transitions between stable and unstable training regimes, and can be easily monitored by the cyclic behavior of the norm of the last layers weights. We empirically observe that without explicit regularization, Grokking as reported in ( arXiv:2201.02177 ) almost exclusively happens at the onset of Slingshots, and is absent without it. While common and easily reproduced in more general settings, the Slingshot Mechanism does not follow from any known optimization theories that we are aware of, and can be easily overlooked without an in depth examination. Our work points to a surprising and useful inductive bias of adaptive gradient optimizers at late stages of training, calling for a revised theoretical analysis of their origin.

Figures (44)

• Figure 1: Slingshot Effects are observed with a fully-connected ReLU network (FCN). The FCN is trained with 200 randomly chosen CIFAR-10 samples with Adam. Multiple Slingshot Effects occur in a cyclic fashion as indicated by the dotted red boxes. Each Slingshot Effect is characterized by a period of rapid growth of the last layer weights, an ensuing training loss spike, and a norm plateau.
• Figure 2: Division dataset: Last layer weight norm growth versus a) loss on training data b) accuracy on training data (c) loss on validation data d) accuracy on validation data e) normalized relative change in features of first Transformer layer (f) normalized relative change in features of second Transformer layer. Note that the feature change plots are shown starting at 10K step to emphasize the feature change behavior during norm growth and plateau phases, revealing that the features stop changing during the norm growth phase and resume changing during the plateaus.
• Figure 3: Curvature metric (denoted as "update sharpness") evolution vs norm growth on (a) addition, (b) subtraction, (c) multiplication, and (d) division dataset. Note the spike in the sharpness metric near the phase transitions between norm growth and plateau.
• Figure 4: Varying $\epsilon$ in Adam on the Division dataset. Observe that as $\epsilon$ increases, there is no Slingshot Effect or grokking behavior. Figure (a) corresponds to default $\epsilon$ suggested in kingma2014adam where the model trained with smallest value undergoes multiple Slingshot cycles.
• Figure 5: Extended analysis on multiple grokking datasets. Points shown in green represent both Slingshot Effects and grokking, points shown blue indicate Slingshot Effects but not grokking while points in red indicate no Slingshot Effects and no grokking. $\epsilon$ in Adam is varied as shown in text. Observe that as $\epsilon$ increases, there are no Slingshot Effects or grokking behavior.