Annealed Leap-Point Sampler for Multimodal Target Distributions

TL;DR

ALPS introduces a scalable framework for sampling high-dimensional multimodal posteriors by augmenting the target with Hessian-informed annealed densities (HAT) and coupling this with a mode-exploration component. The two-core components, ALPS-PA and ALPS-EC, enable non-local mode jumping via a Hessian-based Gaussian mixture proxy and adaptive mode discovery, respectively, while QuanTA-swaps improve temperature-space mixing. Theoretical results show that the coldest temperature level can scale linearly with dimension, $\beta_{\max}=O(d)$, delivering a mixing time of $\mathcal{O}(d)$ for the annealed process and an overall computational cost of $\mathcal{O}(d^3)$; empirically, ALPS demonstrates robust mode discovery and sampling in synthetic, SUR, and spectral-density problems, outperforming traditional parallel tempering and Laplace-approximation-based methods. The work provides a practical, polynomial-time approach to multimodal Bayesian inference in high dimensions with concrete guidance on temperature schedules and mode-finding, and it opens avenues for robustness improvements and extensions to broader model classes.

Abstract

In Bayesian statistics, exploring high-dimensional multimodal posterior distributions poses major challenges for existing MCMC approaches. This paper introduces the Annealed Leap-Point Sampler (ALPS), which augments the target distribution state space with modified annealed (cooled) distributions, in contrast to traditional tempering approaches. The coldest state is chosen such that its annealed density is well-approximated locally by a Laplace approximation. This allows for automated setup of a scalable mode-leaping independence sampler. ALPS requires an exploration component to search for the mode locations, which can either be run adaptively in parallel to improve these mode-jumping proposals, or else as a pre-computation step. A theoretical analysis shows that for a d-dimensional problem the coolest temperature level required only needs to be linear in dimension, $\mathcal{O}\left(d\right)$, implying that the number of iterations needed for ALPS to converge is $\mathcal{O}\left(d\right)$ (typically leading to overall complexity $\mathcal{O}\left(d^3\right)$ when computational cost per iteration is taken into account). ALPS is illustrated on several complex, multimodal distributions that arise from real-world applications. This includes a seemingly-unrelated regression (SUR) model of longitudinal data from U.S. manufacturing firms, as well as a spectral density model that is used in analytical chemistry for identification of molecular biomarkers.

Figures (9)

• Figure 1: Annealing reduces the effect of skew. The dot-dashed line in the bottom panel shows a bimodal density with appreciable skewness; overlayed by the red dashed line representing the Gaussian mixture approximation derived through a local Laplace approximation. The two figures above this show annealed versions of the density and its respective approximation (at annealings $\beta =4$ and $\beta=30$ respectively); demonstrating that as the modes are annealed the deviation from local Gaussianity reduces.
• Figure 2: For the algorithms ALPS (top), LAIS (middle) and PT (bottom) respectively, the trace plots (left) of the first component of the Markov chain as well as marginal density plots (right) of the first 2 components at the target temperature, $\beta=1$. It can be seen from the trace plot (a) that ALPS mixes well and (b) shows that it visits all 13 modes. In contrast, (c) shows that LAIS very rarely jumps between modes and (d) only visits 4 of the modes. From (e) PT appears to be mixing well, but (f) shows that it has only visited 8 modes.
• Figure 3: Empirical acceptance rates of the mode-jumping proposals at the coldest temperature level as a function of dimension, averaged over $128$ runs of $16384$ iterations. For each dimension $d$, the cold inverse temperature is set to $\beta_{\text{max}} = \ell(a)\, d$, where $\ell(a)$ is computed according to \ref{['eq:ell_optim']}. Target acceptance rates $a$ are set to (a) 0.3, (b) 0.5, (c) 0.7 and (d) 0.9.
• Figure 4: (a) 3D perspective plot of the profile likelihood for the bivariate SUR model, adapted from Drton2004; (b) Annealed likelihood at the coldest temperature level used in ALPS.
• Figure 5: Results for ALPS with the bivariate SUR model from Drton2004. The red, dashed lines show the locations of the two modes. Note that the ratio between the hot-state mode finder and the other chains is 1:4, so 236 iterations of the Exploration Component correspond to 944 iterations of the other chains. Therefore, the first 1,000 iterations should be discarded as burn-in.

Theorems & Definitions (50)

• Definition 3.1: Hessian Adjusted Tempered (HAT) distributions
• Definition 3.2: QuanTA Transformation
• Remark 1
• Theorem 5.1: Dimensionality-scaling for the Coldest Temperature Level
• proof
• Remark 2
• Theorem 5.2: Updated dimensionality-scaling for the Coldest Temperature Level
• proof
• Definition A.1: Normalisation at level $\beta$
• Definition A.2: Asymptotic Gaussian standard deviation