EXAdam: The Power of Adaptive Cross-Moments

TL;DR

EXAdam addresses limitations of the Adam optimizer by introducing cross-moment debiasing terms $\tilde{m}$ and $\tilde{v}$ and a gradient-based acceleration term $\tilde{g}$ to produce more adaptive, responsive updates. The cross-moment terms couple the first and second moments with temporal dynamics, while $\tilde{g}$ leverages current gradient information for faster convergence. The paper provides theoretical analysis of these components and demonstrates empirically that EXAdam yields faster convergence and improved accuracies on CIFAR-10 and MinGPT tasks, with only modest computational overhead (~2.5%). Overall, EXAdam aims to offer a more robust, universally applicable optimizer by blending moment-based adaptation with immediate gradient responsiveness, though broader validation remains necessary.

Abstract

This paper introduces EXAdam ($\textbf{EX}$tended $\textbf{Adam}$), a novel optimization algorithm that builds upon the widely-used Adam optimizer. EXAdam incorporates two key enhancements: (1) new debiasing terms for improved moment estimation and (2) a gradient-based acceleration mechanism for increased responsiveness to the current loss landscape. These innovations work synergistically to address limitations of the original Adam algorithm, potentially offering improved convergence properties, enhanced ability to escape saddle points, and potentially greater robustness to hyperparameter choices, though this requires further investigation. We provide a theoretical analysis of EXAdam's components and their interactions, highlighting the algorithm's potential advantages in navigating complex optimization landscapes. Empirical evaluations demonstrate EXAdam's superiority over Adam, achieving 38.46% faster convergence and yielding improvements of 1.96%, 2.17%, and 1.17% in training, validation, and testing accuracies, respectively, when applied to a CNN trained on the CIFAR-10 dataset. While these results are promising, further empirical validation across diverse tasks is essential to fully gauge EXAdam's efficacy. Nevertheless, EXAdam represents a significant advancement in adaptive optimization techniques, with promising implications for a wide range of machine learning applications. This work aims to contribute to the ongoing development of more efficient, adaptive, and universally applicable optimization methods in the field of machine learning and artificial intelligence.

Figures (3)

• Figure 1: The training performance of EXAdam, Adam, AdamW, SGD with momentum, RMSProp, and AdaDelta on the CIFAR-10 dataset. The convexities in the training curves indicate that the ReduceLROnPlateau learning rate scheduler reduced the learning rate.
• Figure 2: The validation performance of EXAdam, Adam, AdamW, SGD with momentum, RMSProp, and AdaDelta on the CIFAR-10 dataset.
• Figure 3: Training loss of MinGPT using EXAdam, Adam, AdamW, AdaFactor, SGD with Momentum, AdEMAMix, and Signum. The loss curves show the convergence behavior of the optimizers during training on the Shakespeare dataset.