Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discretization approximation: An alternative to Monte Carlo in Bayesian computation

Shifeng Xiong

TL;DR

This work introduces discretization approximation (DA) as a deterministic alternative to Monte Carlo methods for Bayesian computation when the posterior density is known up to a normalizing constant. DA discretizes the density on a pre-specified support 𝒜_M⊂[0,1]^d to form a completely known discrete posterior, with convergence to the true posterior governed by the discrepancy δ_{M,d}; quasi-Monte Carlo point sets yield integration-rate convergence that can exceed the standard $O_p(1/√M)$ rate. The paper proves population and sampling-theory results, develops a generalization to unbounded/general regions via a ψ–h transformation, and introduces representation points and two-stage adaptive designs to improve efficiency. Numerical examples across 1D/2D problems, linear regression, and full Bayesian Gaussian process regression demonstrate that DA often outperforms MC/MCMC in accuracy and speed, and can be effectively combined with random Fourier features for scalable GP inference. DA thereby provides a simple, deterministic, and fast complement to traditional Bayesian computation methods with solid theoretical guarantees.

Abstract

In this paper we propose a new deterministic approximation method, called discretization approximation, for Bayesian computation. Discretization approximation is very simple to understand and to implement, It only requires calculating posterior density values as probability masses at pre-specified support points. The resulted discrete distribution can be a good approximation to the target posterior distribution. All posterior quantities, including means, standard deviations, and quantiles, can be approximated by those of this completely known discrete distribution. We establish the convergence rate of discretization approximation as the number of support points goes to infinity. If the support points are generated from quasi-Monte Carlo sequences, then the rate is actually the same as that in integration approximation, generally faster than the optimal statistical rate. In this sense, discretization approximation is superior to the popular Markov chain Monte Carlo method. We also provide random sampling and representation point construction methods from discretization approximation. Numerical examples including some benchmarks demonstrate that the proposed method performs quite well for both low-dimensional and high-dimensional cases.

Discretization approximation: An alternative to Monte Carlo in Bayesian computation

TL;DR

This work introduces discretization approximation (DA) as a deterministic alternative to Monte Carlo methods for Bayesian computation when the posterior density is known up to a normalizing constant. DA discretizes the density on a pre-specified support 𝒜_M⊂[0,1]^d to form a completely known discrete posterior, with convergence to the true posterior governed by the discrepancy δ_{M,d}; quasi-Monte Carlo point sets yield integration-rate convergence that can exceed the standard rate. The paper proves population and sampling-theory results, develops a generalization to unbounded/general regions via a ψ–h transformation, and introduces representation points and two-stage adaptive designs to improve efficiency. Numerical examples across 1D/2D problems, linear regression, and full Bayesian Gaussian process regression demonstrate that DA often outperforms MC/MCMC in accuracy and speed, and can be effectively combined with random Fourier features for scalable GP inference. DA thereby provides a simple, deterministic, and fast complement to traditional Bayesian computation methods with solid theoretical guarantees.

Abstract

In this paper we propose a new deterministic approximation method, called discretization approximation, for Bayesian computation. Discretization approximation is very simple to understand and to implement, It only requires calculating posterior density values as probability masses at pre-specified support points. The resulted discrete distribution can be a good approximation to the target posterior distribution. All posterior quantities, including means, standard deviations, and quantiles, can be approximated by those of this completely known discrete distribution. We establish the convergence rate of discretization approximation as the number of support points goes to infinity. If the support points are generated from quasi-Monte Carlo sequences, then the rate is actually the same as that in integration approximation, generally faster than the optimal statistical rate. In this sense, discretization approximation is superior to the popular Markov chain Monte Carlo method. We also provide random sampling and representation point construction methods from discretization approximation. Numerical examples including some benchmarks demonstrate that the proposed method performs quite well for both low-dimensional and high-dimensional cases.

Paper Structure

This paper contains 11 sections, 3 theorems, 38 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Approximation on the unit hypercube
  3. Population approximation
  4. Sampling properties
  5. Approximation on general regions
  6. Numerical examples
  7. A simple one-dimensional example
  8. Two-dimensional examples
  9. The linear regression
  10. The full Bayesian Gaussian process regression
  11. Discussion

Key Result

Theorem 1

For sufficiently large $M$, we have where $C$ is a positive constant independent of $M$ and $d$. Furthermore, if $E_{(k_1,\ldots,k_d)}(\mu_f)<\infty$ and $V(f_{(k_1,\ldots,k_d)})<\infty$, then for sufficiently large $M$, where $\tilde{C}$ is a positive constant independent of $M$ and $d$.

Figures (3)

  • Figure 1: Comparisons between $\mu_{(f,\mathcal{A}_M)}$ (the solid lines) and $\mu_f$ (the dashed lines), where $f$ is the density of the beta distribution $Beta(2,3)$ and $\mathcal{A}_M=\{(2i-1)/(2M)\}_{i=1}^{M}$ with $M=10$.
  • Figure 2: Contours of the two-dimensional normal density in Section \ref{['subsec:2d']} and the 200 representation points from two-stage DA.
  • Figure 3: Contours of the banana density in Section \ref{['subsec:2d']} and the 200 representation points from two-stage DA.

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2
  • Theorem 3