Fuzzy hyperparameters update in a second order optimization

TL;DR

The paper tackles the inefficiency of traditional first-order optimizers in deep learning by introducing SALO, a second-order optimizer that uses an online diagonal Hessian approximation $H_d$ computed via finite differences to guide weight updates with $\Delta w = - lr \cdot H_d^{-1} \cdot g$. It couples this with a fuzzy logic scheduler that adaptively tunes the learning rate and second-derivative momentum ($\beta_1$, $\beta_3$), reducing sensitivity to hyperparameter choices. Empirical results on TinyImageNet and ImageNet demonstrate SALO’s ability to achieve lower training loss and higher validation accuracy than SGD, Adam, and AdamW, with competitive runtime overhead. The work suggests that combining online curvature information with fuzzy control yields a robust, scalable second-order optimization paradigm for large-scale vision models, and points to future gains from precomputed fuzzy policies and expanded rule sets.

Abstract

This research will present a hybrid approach to accelerate convergence in a second order optimization. An online finite difference approximation of the diagonal Hessian matrix will be introduced, along with fuzzy inferencing of several hyperparameters. Competitive results have been achieved

Figures (5)

• Figure 1: Comparison of our optimizer against ADAM, NewtonCG, BFGS and CG
• Figure 3: Bloc Diagram of Our Fuzzy Scheduling System
• Figure 4: Output Variables
• Figure 5: Training Accuracy: SALO VS. Adam , AdamW and SGD
• Figure 6: Validation Accuracy: SALO VS. Adam , AdamW and SGD