Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autocovariance and Optimal Design for Random Walk Metropolis-Hastings Algorithm

Jingyi Zhang, James C. Spall

TL;DR

The paper investigates the covariance structure of Metropolis-Hastings chains, focusing on scalar MH with symmetric random-walk proposals and symmetric unimodal targets. It derives a closed-form expression for the unit-lag covariance and identifies an optimal symmetric bimodal proposal that minimizes this covariance, thereby enhancing estimation efficiency. In the high-dimensional regime, it connects the covariance structure to the diffusion-limit 0.23 acceptance-rate criterion, providing a principled link between autocovariance and tuning. Numerical experiments across several target families corroborate the theoretical findings, showing improved efficiency for the bimodal proposals and validating the link to known high-dimensional guidelines.

Abstract

The Metropolis-Hastings algorithm has been extensively studied in the estimation and simulation literature, with most prior work focusing on convergence behavior and asymptotic theory. However, its covariance structure-an important statistical property for both theory and implementation-remains less understood. In this work, we provide new theoretical insights into the scalar case, focusing primarily on symmetric unimodal target distributions with symmetric random walk proposals, where we also establish an optimal proposal design. In addition, we derive some more general results beyond this setting. For the high-dimensional case, we relate the covariance matrix to the classical 0.23 average acceptance rate tuning criterion.

Autocovariance and Optimal Design for Random Walk Metropolis-Hastings Algorithm

TL;DR

The paper investigates the covariance structure of Metropolis-Hastings chains, focusing on scalar MH with symmetric random-walk proposals and symmetric unimodal targets. It derives a closed-form expression for the unit-lag covariance and identifies an optimal symmetric bimodal proposal that minimizes this covariance, thereby enhancing estimation efficiency. In the high-dimensional regime, it connects the covariance structure to the diffusion-limit 0.23 acceptance-rate criterion, providing a principled link between autocovariance and tuning. Numerical experiments across several target families corroborate the theoretical findings, showing improved efficiency for the bimodal proposals and validating the link to known high-dimensional guidelines.

Abstract

The Metropolis-Hastings algorithm has been extensively studied in the estimation and simulation literature, with most prior work focusing on convergence behavior and asymptotic theory. However, its covariance structure-an important statistical property for both theory and implementation-remains less understood. In this work, we provide new theoretical insights into the scalar case, focusing primarily on symmetric unimodal target distributions with symmetric random walk proposals, where we also establish an optimal proposal design. In addition, we derive some more general results beyond this setting. For the high-dimensional case, we relate the covariance matrix to the classical 0.23 average acceptance rate tuning criterion.

Paper Structure

This paper contains 9 sections, 7 theorems, 51 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Metropolis-Hastings algorithm
  4. Notation
  5. Problem Setup
  6. Main results
  7. Optimal Proposal Design
  8. Numerical Study
  9. Concluding Remarks and Future Work

Key Result

Lemma 3.1

For a Metropolis-Hastings chain, the general formula for the unit-lag covariance is given by Recall that, unless otherwise specified, all integrals are taken over $\mathbb{R}$.

Figures (3)

  • Figure 1: ESS versus $\mathbb{E}[\alpha(x,y)]$ under matched wall-clock time (2s). Dashed gray shows standard Gaussian proposals; solid lines show bimodal proposals with $\sigma_q/x^\star=0.20,0.40$.
  • Figure 2: Unit-lag correlation decreases with narrower bimodal proposals, approaching the two-point limit—the unattainable minimum under any continuous proposal density.
  • Figure 3: Histograms of MH samples versus target densities for Gaussian $(\mu=0,\sigma=1)$, Logistic $(\mu=3,\sigma=1)$, and GHS $(\alpha=1,\mu=-7,\sigma=1)$. All chains ran for $\approx1$s with bimodal proposal $(\sigma_q/x^\ast=0.40)$.

Theorems & Definitions (15)

  • Lemma 3.1
  • proof
  • Corollary 3.2
  • proof
  • Lemma 3.3
  • proof
  • Theorem 3.4
  • proof
  • Theorem 3.5
  • proof
  • ...and 5 more