Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the stability of the covariance-controlled adaptive Langevin thermostat for large-scale Bayesian sampling

Jiani Wei, Xiaocheng Shang

TL;DR

This paper tackles the stability limits of covariance-controlled adaptive Langevin (CCAdL) thermostats in large-scale Bayesian sampling with noisy gradients. It introduces a Modified CCAdL (mCCAdL) that removes moving-average covariance estimation and solves the additional-term subsystem exactly via a scaling-and-squaring based matrix-exponential approximation, coupled with a symmetric BAODCDOAB splitting to achieve weak second-order convergence. Empirical results across Bayesian linear regression, binary and multiclass classification show that mCCAdL delivers greater numerical stability (enabling larger stepsizes) and higher predictive accuracy than CCAdL, SGHMC, and SGNHT, especially in high-dimensional settings. The work provides a practical, more robust approach to stochastic-gradient MCMC for large datasets, with potential applicability to a range of Bayesian-sampling tasks.

Abstract

Stochastic gradient Langevin dynamics and its variants approximate the likelihood of an entire dataset, via random (and typically much smaller) subsets, in the setting of Bayesian sampling. Due to the (often substantial) improvement of the computational efficiency, they have been widely used in large-scale machine learning applications. It has been demonstrated that the so-called covariance-controlled adaptive Langevin (CCAdL) thermostat, which incorporates an additional term involving the covariance matrix of the noisy force, outperforms popular alternative methods. A moving average is used in CCAdL to estimate the covariance matrix of the noisy force, in which case the covariance matrix will converge to a constant matrix in long-time limit. Moreover, it appears in our numerical experiments that the use of a moving average could reduce the stability of the numerical integrators, thereby limiting the largest usable stepsize. In this article, we propose a modified CCAdL (i.e., mCCAdL) thermostat that uses the scaling part of the scaling and squaring method together with a truncated Taylor series approximation to the exponential to numerically approximate the exact solution to the subsystem involving the additional term proposed in CCAdL. We also propose a symmetric splitting method for mCCAdL, instead of an Euler-type discretisation used in the original CCAdL thermostat. We demonstrate in our numerical experiments that the newly proposed mCCAdL thermostat achieves a substantial improvement in the numerical stability over the original CCAdL thermostat, while significantly outperforming popular alternative stochastic gradient methods in terms of the numerical accuracy for large-scale machine learning applications.

Improving the stability of the covariance-controlled adaptive Langevin thermostat for large-scale Bayesian sampling

TL;DR

This paper tackles the stability limits of covariance-controlled adaptive Langevin (CCAdL) thermostats in large-scale Bayesian sampling with noisy gradients. It introduces a Modified CCAdL (mCCAdL) that removes moving-average covariance estimation and solves the additional-term subsystem exactly via a scaling-and-squaring based matrix-exponential approximation, coupled with a symmetric BAODCDOAB splitting to achieve weak second-order convergence. Empirical results across Bayesian linear regression, binary and multiclass classification show that mCCAdL delivers greater numerical stability (enabling larger stepsizes) and higher predictive accuracy than CCAdL, SGHMC, and SGNHT, especially in high-dimensional settings. The work provides a practical, more robust approach to stochastic-gradient MCMC for large datasets, with potential applicability to a range of Bayesian-sampling tasks.

Abstract

Stochastic gradient Langevin dynamics and its variants approximate the likelihood of an entire dataset, via random (and typically much smaller) subsets, in the setting of Bayesian sampling. Due to the (often substantial) improvement of the computational efficiency, they have been widely used in large-scale machine learning applications. It has been demonstrated that the so-called covariance-controlled adaptive Langevin (CCAdL) thermostat, which incorporates an additional term involving the covariance matrix of the noisy force, outperforms popular alternative methods. A moving average is used in CCAdL to estimate the covariance matrix of the noisy force, in which case the covariance matrix will converge to a constant matrix in long-time limit. Moreover, it appears in our numerical experiments that the use of a moving average could reduce the stability of the numerical integrators, thereby limiting the largest usable stepsize. In this article, we propose a modified CCAdL (i.e., mCCAdL) thermostat that uses the scaling part of the scaling and squaring method together with a truncated Taylor series approximation to the exponential to numerically approximate the exact solution to the subsystem involving the additional term proposed in CCAdL. We also propose a symmetric splitting method for mCCAdL, instead of an Euler-type discretisation used in the original CCAdL thermostat. We demonstrate in our numerical experiments that the newly proposed mCCAdL thermostat achieves a substantial improvement in the numerical stability over the original CCAdL thermostat, while significantly outperforming popular alternative stochastic gradient methods in terms of the numerical accuracy for large-scale machine learning applications.
Paper Structure (14 sections, 31 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 14 sections, 31 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Stochastic gradient
  3. Related work
  4. Covariance-controlled adaptive Langevin thermostat
  5. Contributions
  6. Modified covariance-controlled adaptive Langevin thermostat
  7. Estimating the parameter-dependent covariance matrix
  8. Integrating the subsystem involving the additional term
  9. Splitting methods
  10. Numerical experiments
  11. 2-Wasserstein distance evaluation with Bayesian linear regression
  12. Binary classification with Bayesian logistic regression
  13. Multiclass classification with discriminative restricted Boltzmann machine
  14. Conclusions

Figures (4)

  • Figure 1: Comparisons of the 2-Wasserstein distance of various methods against the number of passes over the entire dataset in the Bayesian linear regression with various values of the stepsize $h$ and effective friction $A$.
  • Figure 2: Comparisons of the test log likelihood of various methods using the posterior mean against the number of passes over the entire dataset in the Bayesian logistic regression on the MNIST dataset of digits 7 and 9 with various values of the stepsize $h$ and effective friction $A$.
  • Figure 3: Comparisons of the test log likelihood of various methods using the posterior mean against the number of passes over the entire dataset in the Bayesian logistic regression on the CIFAR-10 dataset of airplane and automobile (top row), deer and horse (middle row), and cat and dog (bottom row) with various values of the stepsize $h$ and effective friction $A$.
  • Figure 4: Comparisons of the test error rates of various methods using the posterior mean against the number of passes over the entire dataset on datasets letter (top row) and acoustic (bottom row) with various values of the stepsize $h$ and effective friction $A$ indicated.