Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sampling from Constrained Gibbs Measures: with Applications to High-Dimensional Bayesian Inference

Ruixiao Wang, Xiaohong Chen, Sinho Chewi

Abstract

This paper considers a non-standard problem of generating samples from a low-temperature Gibbs distribution with \emph{constrained} support, when some of the coordinates of the mode lie on the boundary. These coordinates are referred to as the non-regular part of the model. We show that in a ``pre-asymptotic'' regime in which the limiting Laplace approximation is not yet valid, the low-temperature Gibbs distribution concentrates on a neighborhood of its mode. Within this region, the distribution is a bounded perturbation of a product measure: a strongly log-concave distribution in the regular part and a one-dimensional exponential-type distribution in each coordinate of the non-regular part. Leveraging this structure, we provide a non-asymptotic sampling guarantee by analyzing the spectral gap of Langevin dynamics. Key examples of low-temperature Gibbs distributions include Bayesian posteriors, and we demonstrate our results on three canonical examples: a high-dimensional logistic regression model, a Poisson linear model, and a Gaussian mixture model.

Sampling from Constrained Gibbs Measures: with Applications to High-Dimensional Bayesian Inference

Abstract

This paper considers a non-standard problem of generating samples from a low-temperature Gibbs distribution with \emph{constrained} support, when some of the coordinates of the mode lie on the boundary. These coordinates are referred to as the non-regular part of the model. We show that in a ``pre-asymptotic'' regime in which the limiting Laplace approximation is not yet valid, the low-temperature Gibbs distribution concentrates on a neighborhood of its mode. Within this region, the distribution is a bounded perturbation of a product measure: a strongly log-concave distribution in the regular part and a one-dimensional exponential-type distribution in each coordinate of the non-regular part. Leveraging this structure, we provide a non-asymptotic sampling guarantee by analyzing the spectral gap of Langevin dynamics. Key examples of low-temperature Gibbs distributions include Bayesian posteriors, and we demonstrate our results on three canonical examples: a high-dimensional logistic regression model, a Poisson linear model, and a Gaussian mixture model.
Paper Structure (68 sections, 31 theorems, 208 equations, 10 figures)

This paper contains 68 sections, 31 theorems, 208 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Non-regular models.
  3. Pre-asymptotic guarantees for MCMC via Poincaré inequalities.
  4. Our contributions.
  5. Related works
  6. Laplace approximation.
  7. Bernstein--von Mises (BvM) theorems.
  8. Non-asymptotic posterior sampling guarantees.
  9. Functional inequalities for low-temperature Gibbs measures.
  10. Notation
  11. Main results
  12. Organization of the paper.
  13. Preliminaries
  14. Sampling from low-temperature Gibbs distributions
  15. Assumptions
  16. ...and 53 more sections

Key Result

Lemma 2.1

Consider a probability distribution $\mu$ with a smooth positive density, and the associated Langevin dynamics where $\{B_t\}_{t\geq 0}$ is a standard Brownian motion. Then, $\mu$ satisfies a Poincaré inequality with constant $C_{\mathsf{PI}}$ if and only if where $X_t \sim \mu_t$.

Figures (10)

  • Figure 1: Effective sample size for each coordinate in logistic regression model, reported for one out of 20 trials with 10000 MCMC steps and step size 0.5.
  • Figure 2: Effective sample size for each coordinate in Poisson linear model, reported for one out of 20 trials with 10000 MCMC steps and step size 0.1.
  • Figure 3: Effective sample size for each coordinate in Gaussian mixture model, reported for one out of 20 trials with 10000 MCMC steps and step size 0.1.
  • Figure 4: LLR bulk ESS across 20 trials for three models.
  • Figure 5: Empirical posterior density of one of the 20 MCMC runs for logistic regression models. The non-regular coordinate $\theta_6$ has a much narrower high-probability region compared to regular coordinates.
  • ...and 5 more figures

Theorems & Definitions (68)

  • Definition 2.1: Poincaré constant
  • Lemma 2.1
  • Lemma 2.2: Tensorization
  • Lemma 2.3: Holley--Stroock perturbation Holley1987LogarithmicSI
  • Remark 3.1
  • Definition 3.1: Good set
  • Remark 3.2: One-sided derivative at the boundary
  • Theorem 3.3: Likelihood decomposition
  • proof
  • Theorem 3.4: Poincaré inequality for the constrained Gibbs measure
  • ...and 58 more