Sampling non-log-concave densities via Hessian-free high-resolution dynamics

TL;DR

The paper develops a non-asymptotic contractivity theory for Hessian-free high-resolution (HFHR) dynamics in sampling from targets $\pi(q) \propto e^{-U(q)}$ with non-convex $U$. By embedding HFHR in a reflection/synchronous coupling framework and using a Lyapunov-weighted Wasserstein distance, it proves exponential contraction for small $\alpha$, matching the invariant Gibbs measure $e^{-U(q)-|p|^2/2}$. Under an additional asymptotically linear-gradient assumption, HFHR is shown to contract faster than kinetic Langevin dynamics, with an explicit linear-in-$\alpha$ gain that applies to both the Lyapunov- and metric-driven regimes. The authors illustrate acceleration through three case studies (multi-well potential, Bayesian linear regression with $L^p$ regularizer, and Bayesian binary classification) and validate the theory with numerical experiments, demonstrating faster convergence and improved performance compared to KLD-based methods.

Abstract

We study the problem of sampling from a target distribution $π(q)\propto e^{-U(q)}$ on $\mathbb{R}^d$, where $U$ can be non-convex, via the Hessian-free high-resolution (HFHR) dynamics, which is a second-order Langevin-type process that has $e^{-U(q)-\frac12|p|^2}$ as its unique invariant distribution, and it reduces to kinetic Langevin dynamics (KLD) as the resolution parameter $α\to0$. The existing theory for HFHR dynamics in the literature is restricted to strongly-convex $U$, although numerical experiments are promising for non-convex settings as well. We focus on studying the convergence of HFHR dynamics when $U$ can be non-convex, which bridges a gap between theory and practice. Under a standard assumption of dissipativity and smoothness on $U$, we adopt the reflection/synchronous coupling method. This yields a Lyapunov-weighted Wasserstein distance in which the HFHR semigroup is exponentially contractive for all sufficiently small $α>0$ whenever KLD is. We further show that, under an additional assumption that asymptotically $\nabla U$ has linear growth at infinity, the contraction rate for HFHR dynamics is strictly better than that of KLD, with an explicit gain. As a case study, we verify the assumptions and the resulting acceleration for three examples: a multi-well potential, Bayesian linear regression with $L^p$ regularizer and Bayesian binary classification. We conduct numerical experiments based on these examples, as well as an additional example of Bayesian logistic regression with real data processed by the neural networks, which illustrates the efficiency of the algorithms based on HFHR dynamics and verifies the acceleration and superior performance compared to KLD.

Figures (4)

• Figure 1: Multi-well potential in dimension $d = 8$.
• Figure 2: Bayesian linear regression.
• Figure 3: Bayesian binary classification.
• Figure 4: Bayesian logistic regression processed by feedforward neural network with $L = 3$ layers.

Theorems & Definitions (103)

• Proposition 2.2: Baseline Lyapunov drift for HFHR dynamics
• proof
• Definition 3.1: Admissible Lyapunov function
• Remark 3.2
• Lemma 3.3: Equivalence of $r$ and the Euclidean distance
• proof
• Remark 3.4
• Lemma 3.5: Drift decomposition
• proof
• Remark 3.6