Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis

TL;DR

AdaFisher introduces a practical second-order optimizer that uses a refined diagonal block-Kronecker approximation of the Fisher Information Matrix to precondition gradients. By focusing on diagonal-dominant Kronecker factors and integrating into an Adam-like adaptive framework, it achieves faster convergence and flatter minima while maintaining computational efficiency suitable for large-scale DNNs. The method is validated across computer vision and natural language tasks, with extensive ablations showing the contributions of FIM-based preconditioning, EMA of factors, and normalization-layer Fisher computations. The results demonstrate improved generalization and robustness to hyperparameter settings, offering a scalable pathway for curvature-aware optimization in real-world deep learning workloads.

Abstract

First-order optimization methods remain the standard for training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by preconditioning the stochastic gradient with a diagonal matrix. Despite the widespread adoption of first-order methods, second-order optimization algorithms often exhibit superior convergence compared to methods like Adam and SGD. However, their practicality in training DNNs is still limited by a significantly higher per-iteration computational cost compared to first-order methods. In this thesis, we present AdaFisher, a novel adaptive second-order optimizer that leverages a diagonal block-Kronecker approximation of the Fisher information matrix to adaptively precondition gradients. AdaFisher aims to bridge the gap between the improved convergence and generalization of second-order methods and the computational efficiency needed for training DNNs. Despite the traditionally slower speed of second-order optimizers, AdaFisher is effective for tasks such as image classification and language modeling, exhibiting remarkable stability and robustness during hyperparameter tuning. We demonstrate that AdaFisher outperforms state-of-the-art optimizers in both accuracy and convergence speed. The code is available from https://github.com/AtlasAnalyticsLab/AdaFisher.

Figures (28)

• Figure 1: Visualizing optimization trajectories for various optimizers overlaid a loss landscape.
• Figure 2: A 2D slice of a loss landscape. This is a 3D rendering of a 2D slice of the loss landscape of a fully-connected network. The slice itself was chosen via gradient descent, using an objective that encouraged this 2D slice to match a target image. Full details are given by lucas2022optimization.
• Figure 3: Three types of stationary points in non-convex optimization landscapes: local minima, global minima, and saddle points.
• Figure 4: Illustration of EFIM computation using K-FAC for a given layer $i$.
• Figure 5: Gershgorin discs and eigenvalue perturbations for the $37$th convolutional layer of ResNet-18 at steps 5200 (middle of training) and 9800 (end of training). Left: Gershgorin discs in the complex plane; Right: Eigenvalue spectrum with and without Gaussian noise added to off-diagonal entries.

Theorems & Definitions (13)

• Definition 2.1.1
• Proposition 3.3.1: EFIM for normalization layer
• Proposition 3.3.2: Efficient EFIM
• Proposition 4.3.1: Convergence in convex optimization
• Proposition 4.3.2: Convergence in non-convex stochastic optimization
• Proposition 1..1: FIM for normalization layer
• proof
• Proposition 1..2: Efficient FIM
• proof
• Proposition 1..3: Convergence in convex optimization