Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

New affine invariant ensemble samplers and their dimensional scaling

Yifan Chen

TL;DR

The paper tackles the challenge of robustly sampling high-dimensional, anisotropic targets by developing affine invariant ensemble MCMC methods. It introduces a derivative-free side move that scales favorably with dimension and a class of derivative-based affine invariant ensemble HMC samplers employing antisymmetric preconditioning, demonstrating $d^{-1/4}$ step-size scaling for Gaussian targets and substantial reductions in autocorrelation times compared to standard HMC. Through Gaussian, ring, and SPDE-inspired experiments, the authors show that affine invariance preserves efficiency across ill-conditioned problems and that gradient-informed ensemble methods can outperform traditional single-chain approaches in high dimensions. The findings highlight the practical potential of affine invariant ensemble samplers for Bayesian inference and PDE-based models, with avenues for adaptive tuning and function-space integration while leaving ergodicity analysis as a future theoretical challenge.

Abstract

We introduce new affine invariant ensemble Markov chain Monte Carlo (MCMC) samplers that are easy to construct and improve upon existing methods, especially for high-dimensional problems. We first propose a simple derivative-free side move sampler that improves upon popular samplers in the \texttt{emcee} package by generating more effective proposal directions. We then develop a class of derivative-based affine invariant ensemble Hamiltonian Monte Carlo (HMC) samplers based on antisymmetric preconditioning using complementary ensembles, which outperform standard, non-affine-invariant HMC when sampling highly anisotropic distributions. We provide asymptotic scaling analysis for high-dimensional Gaussian targets to further elucidate the properties of these affine invariant ensemble samplers. In particular, with derivative information, the affine invariant ensemble HMC can scale much better with dimension compared to derivative-free ensemble samplers.

New affine invariant ensemble samplers and their dimensional scaling

TL;DR

The paper tackles the challenge of robustly sampling high-dimensional, anisotropic targets by developing affine invariant ensemble MCMC methods. It introduces a derivative-free side move that scales favorably with dimension and a class of derivative-based affine invariant ensemble HMC samplers employing antisymmetric preconditioning, demonstrating step-size scaling for Gaussian targets and substantial reductions in autocorrelation times compared to standard HMC. Through Gaussian, ring, and SPDE-inspired experiments, the authors show that affine invariance preserves efficiency across ill-conditioned problems and that gradient-informed ensemble methods can outperform traditional single-chain approaches in high dimensions. The findings highlight the practical potential of affine invariant ensemble samplers for Bayesian inference and PDE-based models, with avenues for adaptive tuning and function-space integration while leaving ergodicity analysis as a future theoretical challenge.

Abstract

We introduce new affine invariant ensemble Markov chain Monte Carlo (MCMC) samplers that are easy to construct and improve upon existing methods, especially for high-dimensional problems. We first propose a simple derivative-free side move sampler that improves upon popular samplers in the \texttt{emcee} package by generating more effective proposal directions. We then develop a class of derivative-based affine invariant ensemble Hamiltonian Monte Carlo (HMC) samplers based on antisymmetric preconditioning using complementary ensembles, which outperform standard, non-affine-invariant HMC when sampling highly anisotropic distributions. We provide asymptotic scaling analysis for high-dimensional Gaussian targets to further elucidate the properties of these affine invariant ensemble samplers. In particular, with derivative information, the affine invariant ensemble HMC can scale much better with dimension compared to derivative-free ensemble samplers.
Paper Structure (34 sections, 2 theorems, 95 equations, 5 figures, 3 tables, 4 algorithms)

This paper contains 34 sections, 2 theorems, 95 equations, 5 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. This work
  3. Derivative-free ensemble samplers
  4. Derivative-based ensemble samplers
  5. Related work on affine invariant samplers
  6. Organization
  7. Affine Invariance
  8. Derivative-free Side Move Sampler
  9. Basic side move
  10. Parallel side move sampler
  11. Comparison to other affine invariant moves
  12. Connection to differential evolution
  13. Analysis of high dimensional scaling
  14. Derivative-based Affine Invariant HMC Samplers
  15. Hamiltonian Monte Carlo
  16. ...and 19 more sections

Key Result

Proposition 3.1

Consider an isotropic Gaussian in $d$ dimensions where $\mathbf{x} \in \mathbb{R}^d$. Under the ideal assumption that $\mathbf{x}_i(m), \mathbf{x}_j(m), \mathbf{x}_k(m)$ are all independent draws from this target distribution, the following holds almost surely.

Figures (5)

  • Figure 1: Demonstration of stretch move and side move. Left: stretch move to the four-pointed star; middle: side move to the five-pointed star; right: both moves when points are on a circle
  • Figure 2: Side move versus stretch move: expected acceptance rate and squared expected jumped distance. Optimal $\alpha$ and $\beta$ in terms of the squared expected jumped distance are marked.
  • Figure 3: Rescaled autocorrelation functions for sampling anisotropic Gaussian targets with condition number $\kappa = 1000$. Left: stretch move; right: side move. Rescaled lag $=$ original lag$/$dim$\times 4$.
  • Figure 4: Autocorrelation time (thinning by $10$) versus dimension for anisotropic Gaussian targets with condition number $\kappa = 1000$. For HMC and affine invariant HMC (Hamiltonian walk and side moves), the total integration time is $T=1$, and the number of leapfrog steps is $n=10$ for HMC while $n=2$ for affine invariant HMC.
  • Figure 5: Autocorrelation time (thinning by $10$) versus dimension for the bimodal SPDE targets. For HMC and affine invariant HMC (Hamiltonian walk and side moves), the total integration time is $T=1$, and the number of leapfrog steps is $n=10$ for HMC while $n=2$ for affine invariant HMC.

Theorems & Definitions (5)

  • Proposition 3.1
  • Remark 3.2
  • Proposition 4.1
  • proof : Proof for Proposition \ref{['prop-gaussian-acceptance']}
  • proof : Proof for Proposition \ref{['thm: scaling-HMC']}