Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kernelized Normalizing Constant Estimation: Bridging Bayesian Quadrature and Bayesian Optimization

Xu Cai, Jonathan Scarlett

TL;DR

This work analyzes the estimation of the normalizing constant $Z = \int_D e^{-\lambda f(\mathbf{x})} d\mathbf{x}$ for functions $f$ in a Matérn RKHS, under zeroth-order queries and potential Gaussian noise. It reveals that the problem interpolates between Bayesian quadrature and Bayesian optimization as $\lambda$ varies, with noise introducing regime-dependent transitions. The authors derive algorithm-independent lower bounds and GP-based two-batch upper bounds, and validate the theory with extensive experiments on analytic, MLP, and PSF tasks, showing pronounced gains of GP-based estimators at low budgets. The results provide a unified theoretical and practical framework bridging BQ and BO for NC estimation, with implications for probabilistic numerical analysis and Bayesian computation in high-dimensional settings.

Abstract

In this paper, we study the problem of estimating the normalizing constant $\int e^{-λf(x)}dx$ through queries to the black-box function $f$, where $f$ belongs to a reproducing kernel Hilbert space (RKHS), and $λ$ is a problem parameter. We show that to estimate the normalizing constant within a small relative error, the level of difficulty depends on the value of $λ$: When $λ$ approaches zero, the problem is similar to Bayesian quadrature (BQ), while when $λ$ approaches infinity, the problem is similar to Bayesian optimization (BO). More generally, the problem varies between BQ and BO. We find that this pattern holds true even when the function evaluations are noisy, bringing new aspects to this topic. Our findings are supported by both algorithm-independent lower bounds and algorithmic upper bounds, as well as simulation studies conducted on a variety of benchmark functions.

Kernelized Normalizing Constant Estimation: Bridging Bayesian Quadrature and Bayesian Optimization

TL;DR

This work analyzes the estimation of the normalizing constant for functions in a Matérn RKHS, under zeroth-order queries and potential Gaussian noise. It reveals that the problem interpolates between Bayesian quadrature and Bayesian optimization as varies, with noise introducing regime-dependent transitions. The authors derive algorithm-independent lower bounds and GP-based two-batch upper bounds, and validate the theory with extensive experiments on analytic, MLP, and PSF tasks, showing pronounced gains of GP-based estimators at low budgets. The results provide a unified theoretical and practical framework bridging BQ and BO for NC estimation, with implications for probabilistic numerical analysis and Bayesian computation in high-dimensional settings.

Abstract

In this paper, we study the problem of estimating the normalizing constant through queries to the black-box function , where belongs to a reproducing kernel Hilbert space (RKHS), and is a problem parameter. We show that to estimate the normalizing constant within a small relative error, the level of difficulty depends on the value of : When approaches zero, the problem is similar to Bayesian quadrature (BQ), while when approaches infinity, the problem is similar to Bayesian optimization (BO). More generally, the problem varies between BQ and BO. We find that this pattern holds true even when the function evaluations are noisy, bringing new aspects to this topic. Our findings are supported by both algorithm-independent lower bounds and algorithmic upper bounds, as well as simulation studies conducted on a variety of benchmark functions.
Paper Structure (28 sections, 9 theorems, 50 equations, 6 figures, 1 table, 1 algorithm)

This paper contains 28 sections, 9 theorems, 50 equations, 6 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setup
  4. Lower Bounds
  5. Upper Bounds
  6. Experiments
  7. Setup
  8. Analytic Functions
  9. Multi-layer Perception (MLP)
  10. Point Spread Function (PSF)
  11. Discussion
  12. Effect of Varying $\lambda$
  13. Conclusion
  14. Proofs of Lower Bounds
  15. Function Class Construction
  16. ...and 13 more sections

Key Result

Theorem 1

(Noiseless Lower Bound) Consider the noiseless setting with $f\in\mathcal{H}^{\nu}(B)$, $\nu+\frac{d}{2}\ge 1$, and a time horizon $T\to \infty$. For any algorithm that estimates $Z$ and produces an estimate $\hat{Z}$ satisfying eq:error, the worst-case $f\in\mathcal{H}^{\nu}(B)$ must have the foll

Figures (6)

  • Figure 1: An illustration of estimating the normalizing constant under small (0.1) and large (10) $\lambda$.
  • Figure 2: Results for analytic functions.
  • Figure 3: Results for MLP.
  • Figure 4: Results for estimating PSF.
  • Figure 5: Additional results for MLP.
  • ...and 1 more figures

Theorems & Definitions (10)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • Lemma 5