Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix

Mrinmay Sen, A. K. Qin, Gayathri C, Raghu Kishore N, Yen-Wei Chen, Balasubramanian Raman

TL;DR

SOFIM tackles the inefficiency of Newton-style optimization in large-scale stochastic settings by using a regularized Fisher information matrix as a surrogate Hessian and computing updates with a Sherman–Morrison-based inverse, achieving fast convergence with the same order of memory and time as SGD with momentum. It integrates Adam-like first-moment bias correction to stabilize updates under data non-stationarity. The key contributions include the regularized FIM formulation, a closed-form Newton update via rank-1 updates, and convergence guarantees for convex losses, demonstrated through extensive experiments on standard image classification datasets. The practical impact is a robust, scalable second-order-like optimizer that outperforms SGD variants and several Newton-based methods while maintaining low computational overhead. The approach holds promise for broad applicability to deep learning and probabilistic modeling tasks that benefit from curvature-aware updates.

Abstract

This paper introduces a new stochastic optimization method based on the regularized Fisher information matrix (FIM), named SOFIM, which can efficiently utilize the FIM to approximate the Hessian matrix for finding Newton's gradient update in large-scale stochastic optimization of machine learning models. It can be viewed as a variant of natural gradient descent, where the challenge of storing and calculating the full FIM is addressed through making use of the regularized FIM and directly finding the gradient update direction via Sherman-Morrison matrix inversion. Additionally, like the popular Adam method, SOFIM uses the first moment of the gradient to address the issue of non-stationary objectives across mini-batches due to heterogeneous data. The utilization of the regularized FIM and Sherman-Morrison matrix inversion leads to the improved convergence rate with the same space and time complexities as stochastic gradient descent (SGD) with momentum. The extensive experiments on training deep learning models using several benchmark image classification datasets demonstrate that the proposed SOFIM outperforms SGD with momentum and several state-of-the-art Newton optimization methods in term of the convergence speed for achieving the pre-specified objectives of training and test losses as well as test accuracy.

SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix

TL;DR

SOFIM tackles the inefficiency of Newton-style optimization in large-scale stochastic settings by using a regularized Fisher information matrix as a surrogate Hessian and computing updates with a Sherman–Morrison-based inverse, achieving fast convergence with the same order of memory and time as SGD with momentum. It integrates Adam-like first-moment bias correction to stabilize updates under data non-stationarity. The key contributions include the regularized FIM formulation, a closed-form Newton update via rank-1 updates, and convergence guarantees for convex losses, demonstrated through extensive experiments on standard image classification datasets. The practical impact is a robust, scalable second-order-like optimizer that outperforms SGD variants and several Newton-based methods while maintaining low computational overhead. The approach holds promise for broad applicability to deep learning and probabilistic modeling tasks that benefit from curvature-aware updates.

Abstract

This paper introduces a new stochastic optimization method based on the regularized Fisher information matrix (FIM), named SOFIM, which can efficiently utilize the FIM to approximate the Hessian matrix for finding Newton's gradient update in large-scale stochastic optimization of machine learning models. It can be viewed as a variant of natural gradient descent, where the challenge of storing and calculating the full FIM is addressed through making use of the regularized FIM and directly finding the gradient update direction via Sherman-Morrison matrix inversion. Additionally, like the popular Adam method, SOFIM uses the first moment of the gradient to address the issue of non-stationary objectives across mini-batches due to heterogeneous data. The utilization of the regularized FIM and Sherman-Morrison matrix inversion leads to the improved convergence rate with the same space and time complexities as stochastic gradient descent (SGD) with momentum. The extensive experiments on training deep learning models using several benchmark image classification datasets demonstrate that the proposed SOFIM outperforms SGD with momentum and several state-of-the-art Newton optimization methods in term of the convergence speed for achieving the pre-specified objectives of training and test losses as well as test accuracy.
Paper Structure (14 sections, 9 equations, 9 figures, 1 algorithm)

This paper contains 14 sections, 9 equations, 9 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Fisher information matrix (FIM)
  4. Natural gradient descent (NGD)
  5. Sherman Morrison formula of matrix inversion
  6. Proposed method
  7. Regularized fisher information matrix (FIM)
  8. Calculate Newton update using regularized FIM
  9. Complexities
  10. Convergence guarantee for convex loss function
  11. Experimental setup
  12. Results
  13. Effect of $\rho$
  14. Conclusions and Future work

Figures (9)

  • Figure 1: Iteration-wise comparisons of various methods regarding training loss, test loss, and test accuracy on CIFAR10 using the LeNet5 model.
  • Figure 2: Iteration-wise comparisons of various methods regarding training loss, test loss, and test accuracy on CIFAR100 using the Resnet9 model.
  • Figure 3: Iteration-wise comparisons of various methods regarding training loss, test loss, and test accuracy on SVHN using the Resnet9 model
  • Figure 4: Iteration-wise comparisons of various methods regarding training loss, test loss, and test accuracy on CIFAR10 using the Resnet18 model
  • Figure 5: Time comparisons of various methods regarding training loss, test loss, and test accuracy on CIFAR10 using the LeNet5 model.
  • ...and 4 more figures