Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational Stochastic Gradient Descent for Deep Neural Networks

Haotian Chen, Anna Kuzina, Babak Esmaeili, Jakub M Tomczak

TL;DR

The paper addresses the gap between traditional adaptive gradient optimizers and probabilistic gradient modeling by introducing Variational Stochastic Gradient Descent (VSGD). It formulates gradient updates as a probabilistic inference problem and derives update rules via stochastic variational inference, supplemented by a kernel-smoothing interpretation and a conjugate-exponential-family framework. Key contributions include closed-form local and global parameter updates, a tunable prior strength $\gamma$ that modulates information from priors versus observations, and empirical advantages over Adam and SGD on multiple image-classification tasks and architectures. This work offers a principled optimizer for deep networks that can better handle gradient uncertainty and potentially improve generalization and robustness in practice.

Abstract

Current state-of-the-art optimizers are adaptive gradient-based optimization methods such as Adam. Recently, there has been an increasing interest in formulating gradient-based optimizers in a probabilistic framework for better modeling the uncertainty of the gradients. Here, we propose to combine both approaches, resulting in the Variational Stochastic Gradient Descent (VSGD) optimizer. We model gradient updates as a probabilistic model and utilize stochastic variational inference (SVI) to derive an efficient and effective update rule. Further, we show how our VSGD method relates to other adaptive gradient-based optimizers like Adam. Lastly, we carry out experiments on two image classification datasets and four deep neural network architectures, where we show that VSGD outperforms Adam and SGD.

Variational Stochastic Gradient Descent for Deep Neural Networks

TL;DR

The paper addresses the gap between traditional adaptive gradient optimizers and probabilistic gradient modeling by introducing Variational Stochastic Gradient Descent (VSGD). It formulates gradient updates as a probabilistic inference problem and derives update rules via stochastic variational inference, supplemented by a kernel-smoothing interpretation and a conjugate-exponential-family framework. Key contributions include closed-form local and global parameter updates, a tunable prior strength that modulates information from priors versus observations, and empirical advantages over Adam and SGD on multiple image-classification tasks and architectures. This work offers a principled optimizer for deep networks that can better handle gradient uncertainty and potentially improve generalization and robustness in practice.

Abstract

Current state-of-the-art optimizers are adaptive gradient-based optimization methods such as Adam. Recently, there has been an increasing interest in formulating gradient-based optimizers in a probabilistic framework for better modeling the uncertainty of the gradients. Here, we propose to combine both approaches, resulting in the Variational Stochastic Gradient Descent (VSGD) optimizer. We model gradient updates as a probabilistic model and utilize stochastic variational inference (SVI) to derive an efficient and effective update rule. Further, we show how our VSGD method relates to other adaptive gradient-based optimizers like Adam. Lastly, we carry out experiments on two image classification datasets and four deep neural network architectures, where we show that VSGD outperforms Adam and SGD.
Paper Structure (17 sections, 7 equations, 4 figures, 4 tables)

This paper contains 17 sections, 7 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background knowledge
  3. Conclusion
  4. Derivations
  5. Derivation of VSGD
  6. A Kernel Smoothing View on VSGD
  7. Treating Samples Separately in a Mini-bach
  8. Experimental Details
  9. Architecture
  10. VGG
  11. ConvMixer
  12. ResNeXt
  13. Hyperparameters
  14. Data Augmentations
  15. Training Loss
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Training loss on CIFAR100 dataset.
  • Figure 4: Analysis of the hyperparameter $\gamma$'sensitivity on the VSGD's performance in terms of accuracy.
  • Figure 5: CIFAR100 learning curves trained with ConvMixer architecture for different $\gamma$ values.
  • Figure 6: Comparison of Constant-VSGD and VSGD learning curves trained on CIFAR100 with ConvMixer architecture. The final accuracy for VSGD is $69.04 \pm 0.4$, while the accuracy for Constant-VSGD is $68.49 \pm 0.19$.