ZENITH: Automated Gradient Norm Informed Stochastic Optimization

TL;DR

ZENITH introduces a gradient-norm informed learning-rate scheduler that ties the LR to the historical peak of gradient norms via η_t = η_0 · (μ_t / Z_t), computed from a sliding window of recent gradient magnitudes. This approach yields scale-invariant, regularization-friendly LR adaptation with negligible memory overhead and only two hyperparameters (η_0 and W). Across six CNN architectures on six vision benchmarks and MS COCO detection/segmentation tasks, ZENITH achieves higher test accuracy and reduced wall-clock time compared to baselines, supported by analyses linking its dynamics to convergence toward flatter minima. The practical impact is a robust, efficient LR strategy that minimizes tuning and hardware requirements while maintaining strong generalization.

Abstract

Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule. While existing adaptive optimizers schedule the LR automatically, they suffer from computational and memory overhead, incompatibility with regularization, and suboptimal LR choices. In this work, we introduce the ZENITH (Zero-overhead Evolution using Norm-Informed Training History) optimizer, which adapts the LR using the temporal evolution of the gradient norm. Image classification experiments spanning 6 CNN architectures and 6 benchmarks demonstrate that ZENITH achieves higher test accuracy in lower wall-clock time than baselines. It also yielded superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO using the R-CNN family of models. Furthermore, its compatibility with regularization enables even better generalization.

Figures (6)

• Figure 1: Training loss and test accuracy curves against wall-clock time for the 6 image classification experiments. The color-coding for each optimizer is as follows: ALIG, COCOB, D-Adaptation, DoG, DoWG, GeN, L4, LQA, PAL, Prodigy, SPS, and ZENITH. This color scheme is consistent across all other figures as well. Training loss magnitudes in this figure are scaled by a factor of 10 or 100 for visual clarity. This linear transformation preserves the original curve profiles without any distortions. The loss function was not scaled during the actual training process. Additionally, all curves are smoothed using a simple moving average for visual clarity.
• Figure 2: (a) Evolution of the $L_2$ gradient norm and (b) the corresponding LR used by ZENITH across iterations.
• Figure 3: (a–b) Influence of the LR on minima sharpness and test accuracy for vanilla SGD; (c) conceptual explanation of the generalization benefits of flat minima; (d) loss landscapes around the minima in the CIFAR-100 experiment; (e) accuracy versus minima sharpness across 4 experiments; (f) LR trajectories for distance-aware and Polyak-style baselines; (g–h) sensitivity of optimizers to the choice of initial LR.
• Figure 4: Impact of regularization of strength 2e-4 on LR trajectories in (a) CIFAR-100 and (b) Food-101.
• Figure 5: Training loss and test mAP50 curves against wall-clock time for detection and segmentation experiments. The color-coding for each optimizer is as follows: COCOB, D-Adaptation, DoG, PAL, and ZENITH. Training loss magnitudes in this figure are scaled by a factor of 10 or 100 for visual clarity. This linear transformation preserves the original curve profiles without any distortions. The loss function was not scaled during the actual training process. Curves are also smoothed using a simple moving average for visual clarity.

Theorems & Definitions (4)

• Definition 2.1: ZENITH Algorithm
• Proposition 2.2
• Theorem 2.5
• proof