Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information

Dongseong Hwang

TL;DR

The modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.

Abstract

This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We provide an accessible and detailed analysis of the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, adaptive epsilon, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.

FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information

TL;DR

The modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.

Abstract

This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We provide an accessible and detailed analysis of the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, adaptive epsilon, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.
Paper Structure (39 sections, 42 equations, 1 figure, 6 tables, 4 algorithms)

This paper contains 39 sections, 42 equations, 1 figure, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Gradient on Riemannian manifold
  4. Fisher Information Matrix (FIM) and Natural Gradient
  5. Toward FAdam
  6. Loss and FIM
  7. Diagonal Fisher Information
  8. Empirical Fisher Information
  9. Discrete vs Continuous probability distributions
  10. Fisher Adam
  11. Reciprocal vs Reciprocal Square-root for FIM
  12. Momentum
  13. Weight decay on manifold
  14. FAdam
  15. Convergence analysis
  16. ...and 24 more sections

Figures (1)

  • Figure 1: Comparison of FAdam and Adam performance. (a) Eval loss (log pplx) on 1B LLMs, presenting FAdafactor outperforms Adafactor. (b) Average WER on LibriSpeech using 600M Conformer models, presenting FAdam outperforms Adam. (c) FID of ImageNet generation using 100M VQ-VAE models, presenting FAdam outperforms AdamW. (d) Comparison of FIM exponents on a 1B LLM, showing 0.5 (square root) as the optimal choice.