Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

Chunyuan Li, Changyou Chen, David Carlson, Lawrence Carin

TL;DR

<3-5 sentences> The paper tackles two core challenges in training deep neural networks: difficult curvature and overfitting. It introduces Preconditioned Stochastic Gradient Langevin Dynamics (pSGLD), which combines a RMSprop-style diagonal preconditioner $G(\boldsymbol{\theta})$ with SGLD to adapt updates to the local geometry of the parameter space, providing theoretical convergence guarantees and finite-time error bounds. Empirically, pSGLD yields faster convergence, higher effective sample size, and state-of-the-art predictive performance on Bayesian logistic regression, feedforward networks, and convolutional nets (e.g., MNIST, CIFAR-10) while offering scalable uncertainty estimates. Practical techniques, such as omitting the $\Gamma(\boldsymbol{\theta}_t)$ term and sample thinning, further improve efficiency without sacrificing accuracy.

Abstract

Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to the local geometry of parameter space. A second issue is overfitting, which is typically addressed by early stopping. However, recent work has demonstrated that Bayesian model averaging mitigates this problem. The posterior can be sampled by using Stochastic Gradient Langevin Dynamics (SGLD). However, the rapidly changing curvature renders default SGLD methods inefficient. Here, we propose combining adaptive preconditioners with SGLD. In support of this idea, we give theoretical properties on asymptotic convergence and predictive risk. We also provide empirical results for Logistic Regression, Feedforward Neural Nets, and Convolutional Neural Nets, demonstrating that our preconditioned SGLD method gives state-of-the-art performance on these models.

Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

TL;DR

<3-5 sentences> The paper tackles two core challenges in training deep neural networks: difficult curvature and overfitting. It introduces Preconditioned Stochastic Gradient Langevin Dynamics (pSGLD), which combines a RMSprop-style diagonal preconditioner with SGLD to adapt updates to the local geometry of the parameter space, providing theoretical convergence guarantees and finite-time error bounds. Empirically, pSGLD yields faster convergence, higher effective sample size, and state-of-the-art predictive performance on Bayesian logistic regression, feedforward networks, and convolutional nets (e.g., MNIST, CIFAR-10) while offering scalable uncertainty estimates. Practical techniques, such as omitting the term and sample thinning, further improve efficiency without sacrificing accuracy.

Abstract

Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to the local geometry of parameter space. A second issue is overfitting, which is typically addressed by early stopping. However, recent work has demonstrated that Bayesian model averaging mitigates this problem. The posterior can be sampled by using Stochastic Gradient Langevin Dynamics (SGLD). However, the rapidly changing curvature renders default SGLD methods inefficient. Here, we propose combining adaptive preconditioners with SGLD. In support of this idea, we give theoretical properties on asymptotic convergence and predictive risk. We also provide empirical results for Logistic Regression, Feedforward Neural Nets, and Convolutional Neural Nets, demonstrating that our preconditioned SGLD method gives state-of-the-art performance on these models.

Paper Structure

This paper contains 27 sections, 4 theorems, 11 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Preconditioned Stochastic Gradient Langevin Dynamics
  5. Preconditioned SGLD Algorithms in Practice
  6. Finite-time Error Analysis
  7. Practical Techniques
  8. Excluding $\Gamma(\boldsymbol{\theta}_t)$ term
  9. Thinning samples
  10. Experiments
  11. Simulation
  12. Bayesian Logstic Regression
  13. Feedforward Neural Networks
  14. Convolutional Neural Networks
  15. Conclusion
  16. ...and 12 more sections

Key Result

Theorem 1

Define the operator $\Delta V_t = (N \bar{g}(\boldsymbol{\theta}_t; \mathcal{D}^t) - g(\boldsymbol{\theta}_t; \mathcal{D}^t))^\top G^{} (\boldsymbol{\theta}_t) \nabla_{\boldsymbol{\theta}}$. Under Assumption ass:stepsize_constraints, for a test function $\phi$, the MSE of the pSGLD at finite time $S

Figures (9)

  • Figure 1: Simulation results on a 2D Gaussian.
  • Figure 2: BLR on Australian dataset.
  • Figure 3: Learning curves.
  • Figure 4: FNN of size 1200-1200 on MNIST.
  • Figure 5: Learning curves.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Corollary 2
  • Corollary 3
  • Lemma 4