Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybrid Geometry-Adaptive MCMC for Bayesian Inference in Higher-Order Ising Models

Godwin Osabutey, Robert Richardson, Garritt L. Page

TL;DR

The paper tackles the Bayesian inverse problem for a mean-field Ising model augmented with three-body interactions, where standard estimators struggle near criticality and under non-identifiability. It derives a low-dimensional representation of the partition function via magnetization and uses this to form a tractable thermodynamic limit, including a mean-field self-consistency equation and a compact log-likelihood gradient. To address difficult posterior geometry, it develops a hybrid MCMC that alternates geometry-aware RMHMC steps with adaptive Metropolis-Hastings updates, aided by grid-based initialization and a regularized Fisher-information-based metric to ensure stable sampling. Through simulations across bimodal, unimodal, and non-identifiable regimes, the hybrid sampler achieves faster convergence, better mixing, and reliable uncertainty quantification, even when individual parameters remain weakly identified while observable distributions remain accurate. These methods have immediate relevance for learning higher-order energy-based models and Boltzmann machines, providing principled uncertainty quantification and practical strategies to diagnose identifiability and stability in complex graphical models.

Abstract

We address the inverse problem for the mean-field Ising model with two- and three-body interactions using a Bayesian framework. Parameter recovery in this setting is notoriously difficult, particularly near phase transitions, at criticality, and under non-identifiability, where conventional estimators and standard MCMC samplers fail. To overcome these challenges, we develop a hybrid algorithm that combines Adaptive Metropolis Hastings with geometry-aware Riemannian manifold Hamiltonian dynamics. This approach yields substantially improved mixing and convergence in the three-dimensional parameter space. Through simulated experiments across representative regimes, we demonstrate that the method achieves accurate density reconstruction and reliable uncertainty quantification even in settings where existing approaches are unstable or inapplicable.

Hybrid Geometry-Adaptive MCMC for Bayesian Inference in Higher-Order Ising Models

TL;DR

The paper tackles the Bayesian inverse problem for a mean-field Ising model augmented with three-body interactions, where standard estimators struggle near criticality and under non-identifiability. It derives a low-dimensional representation of the partition function via magnetization and uses this to form a tractable thermodynamic limit, including a mean-field self-consistency equation and a compact log-likelihood gradient. To address difficult posterior geometry, it develops a hybrid MCMC that alternates geometry-aware RMHMC steps with adaptive Metropolis-Hastings updates, aided by grid-based initialization and a regularized Fisher-information-based metric to ensure stable sampling. Through simulations across bimodal, unimodal, and non-identifiable regimes, the hybrid sampler achieves faster convergence, better mixing, and reliable uncertainty quantification, even when individual parameters remain weakly identified while observable distributions remain accurate. These methods have immediate relevance for learning higher-order energy-based models and Boltzmann machines, providing principled uncertainty quantification and practical strategies to diagnose identifiability and stability in complex graphical models.

Abstract

We address the inverse problem for the mean-field Ising model with two- and three-body interactions using a Bayesian framework. Parameter recovery in this setting is notoriously difficult, particularly near phase transitions, at criticality, and under non-identifiability, where conventional estimators and standard MCMC samplers fail. To overcome these challenges, we develop a hybrid algorithm that combines Adaptive Metropolis Hastings with geometry-aware Riemannian manifold Hamiltonian dynamics. This approach yields substantially improved mixing and convergence in the three-dimensional parameter space. Through simulated experiments across representative regimes, we demonstrate that the method achieves accurate density reconstruction and reliable uncertainty quantification even in settings where existing approaches are unstable or inapplicable.
Paper Structure (15 sections, 2 theorems, 28 equations, 8 figures, 2 tables, 1 algorithm)

This paper contains 15 sections, 2 theorems, 28 equations, 8 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The model and methods
  3. The mean-field Ising model
  4. Bayesian formulation
  5. Identifiability
  6. Markov Chain Monte Carlo Algorithm
  7. Computational considerations
  8. Simulations Based on Challenging (K,J,h)
  9. Case 1: Bimodal Likelihood Surface
  10. Case 2: Unimodal Density
  11. Case 3: Non-identifiable Likelihood Surface
  12. Convergence Measures
  13. Simulation Study
  14. Conclusion
  15. AMH and RMAHMC: non-identifiable case

Key Result

Corollary 2.1

Let $S_m=\{-1+\tfrac{2k}{N}: k=0,\dots,N\}$ denote the possible values of $m_N(\mathbf{x})$. Then where $A_N(m)=\binom{N}{\frac{1+m}{2}N} = \text{card}\{\mathbf{x}\in\{-1,1\}^N : m_N(\mathbf{x})=m\}$ counts the microstates corresponding to a given macrostate $m$. Here $\beta$ is absorbed into $\boldsymbol{\theta}$.

Figures (8)

  • Figure 1: Plot (a) and (b) displays the log-likelihood as a function of $K$ and $J$ while keeping $h$ fixed. The white 'x' marker indicates the specific points (i.e., $(0, 1.2, 0)$ and $(1.67,0.01,0.1)$) in the parameter space we aim to recover. The gray phase corresponds to areas in the parameter space where the likelihood is numerically zero.
  • Figure 2: Trace plot of the MCMC samples collected from the posterior distribution of $(K,J,h)$ based on $N = 300$ and $M=1000$ sampled configurations using $(K,J,h)=(0,1.2,0)$ (first row) and $(K,J,h)=(1.67,0.01,0.1)$ (second row).
  • Figure 3: Boltzmann-Gibbs distribution of $m_N(\mathbf{x})$ at $N = 300$ with $M = 1000$ for the true and estimated parameters at (a) $(K,J,h) =(0,1.2,0)$ and (b) $(K,J,h)=(1.67,0.01,0.1)$. In (a), the peaks of the distribution are centered around two distinct and opposite values of $m_N(\mathbf{x})$ such that $\mathbb{E}m_N(\mathbf{x})=0$. The two peaks corresponds to global stable states of the system. The larger peak in (b) corresponds to a metastable state with probability approaching $0$ as $N$ increases. The red curve corresponds to the density plot estimated using MCMC samples collected based on the RMAHMC-AMH algorithm.
  • Figure 4: Plot (a) displays the log-likelihood as a function of $K$ and $J$ for fixed $h=0.1$. Plot (b) displays the log-likelihood as a function of $K$ and $J$ while keeping $h$ fixed at 0. The white 'x' marker indicates the point in the parameter space we aim to recover. The gray phase corresponds to areas in the parameter space where the likelihood is numerically zero and the red area depicts the flat regions of the likelihood function.
  • Figure 5: Trace plots associated with MCMC samples from the posterior distribution of $(K,J,h)$ based on $N = 300$ and $M=1000$ sampled configurations. The first row displays the three MCMC algorithms for $(K,J,h)=(0.5,0.3,0.1)$ and the second row for $(K,J,h)=(0,1,0)$.
  • ...and 3 more figures

Theorems & Definitions (4)

  • Corollary 2.1
  • Definition 3.1: Coarse-grid initialization
  • Lemma 3.1: SPD preservation
  • proof : Proof.