Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks

TL;DR

Frankenstein addresses the generalization gap observed with adaptive optimizers by dynamically adjusting momentum and learning dynamics. It couples an adaptive first-moment coefficient $β$, a relaxation strategy with a dynamic EMA for the second moment $v_t$ via a variable $β_2$, and an acceleration factor $ξ$ to reflect gradient changes. Across vision, NLP, few-shot learning, and materials simulations, Frankenstein achieves faster convergence and better generalization than Adam-like optimizers and SGD, as demonstrated by extensive experiments and qualitative analyses. Centered Kernel Alignment (CKA) and loss-landscape visualizations provide mechanistic insights into the training dynamics, supporting the optimizer’s broad applicability and practical impact.

Abstract

Gradient-based optimization drives the unprecedented performance of modern deep neural network models across diverse applications. Adaptive algorithms have accelerated neural network training due to their rapid convergence rates; however, they struggle to find ``flat minima" reliably, resulting in suboptimal generalization compared to stochastic gradient descent (SGD). By revisiting various adaptive algorithms' mechanisms, we propose the Frankenstein optimizer, which combines their advantages. The proposed Frankenstein dynamically adjusts first- and second-momentum coefficients according to the optimizer's current state to directly maintain consistent learning dynamics and immediately reflect sudden gradient changes. Extensive experiments across several research domains such as computer vision, natural language processing, few-shot learning, and scientific simulations show that Frankenstein surpasses existing adaptive algorithms and SGD empirically regarding convergence speed and generalization performance. Furthermore, this research deepens our understanding of adaptive algorithms through centered kernel alignment analysis and loss landscape visualization during the learning process. Code is available at https://github.com/acctouhou/Frankenstein_optimizer

Figures (13)

• Figure 1: From top to bottom: (1) Adaptive factor for each optimizer during the optimization process of f(x); (2) Function f(x) and its corresponding gradient; (3) Historical trajectories of each optimizer as a function of optimization steps.
• Figure 2: Testing set accuracy profiles of various optimizers on EfficientNetB0 with CIFAR-100 (a) and (b). (c) Testing set accuracy according to architectures (EfficientNetB0-6).
• Figure 3: Training, validation and testing set accuracy with different optimizers deal with IMDB movie reviews using BERT.
• Figure 4: Training and testing losses for text lengths (a) 256 and (b) 512 from the instruction-tuning process of the LLaMA 7B model.
• Figure 5: The testing result of several optimizers on the Model-Agnostic Meta-Learning task with multiple strategies.