Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fusion of Gaussian Processes Predictions with Monte Carlo Sampling

Marzieh Ajirak, Daniel Waxman, Fernando Llorente, Petar M. Djuric

TL;DR

This work addresses the problem of fusing predictive pdfs from multiple Gaussian process models within a Bayesian framework. It develops linear and novel log-linear pooling approaches, with input-dependent weights learned via Monte Carlo sampling, using random Fourier feature approximations for scalability. The key contributions are two log-linear fusion variants, Product-BHS (P-BHS) and PoGPE, alongside a principled Bayesian treatment of weight learning and joint model inference, demonstrated on synthetic data with NLPD as the performance metric. The results show that log-linear pooling with unconstrained weights can outperform traditional linear stacking, especially as the number of experts or the feature richness increases, offering a principled and scalable approach for GP ensemble fusion in scientific modeling.

Abstract

In science and engineering, we often work with models designed for accurate prediction of variables of interest. Recognizing that these models are approximations of reality, it becomes desirable to apply multiple models to the same data and integrate their outcomes. In this paper, we operate within the Bayesian paradigm, relying on Gaussian processes as our models. These models generate predictive probability density functions (pdfs), and the objective is to integrate them systematically, employing both linear and log-linear pooling. We introduce novel approaches for log-linear pooling, determining input-dependent weights for the predictive pdfs of the Gaussian processes. The aggregation of the pdfs is realized through Monte Carlo sampling, drawing samples of weights from their posterior. The performance of these methods, as well as those based on linear pooling, is demonstrated using a synthetic dataset.

Fusion of Gaussian Processes Predictions with Monte Carlo Sampling

TL;DR

This work addresses the problem of fusing predictive pdfs from multiple Gaussian process models within a Bayesian framework. It develops linear and novel log-linear pooling approaches, with input-dependent weights learned via Monte Carlo sampling, using random Fourier feature approximations for scalability. The key contributions are two log-linear fusion variants, Product-BHS (P-BHS) and PoGPE, alongside a principled Bayesian treatment of weight learning and joint model inference, demonstrated on synthetic data with NLPD as the performance metric. The results show that log-linear pooling with unconstrained weights can outperform traditional linear stacking, especially as the number of experts or the feature richness increases, offering a principled and scalable approach for GP ensemble fusion in scientific modeling.

Abstract

In science and engineering, we often work with models designed for accurate prediction of variables of interest. Recognizing that these models are approximations of reality, it becomes desirable to apply multiple models to the same data and integrate their outcomes. In this paper, we operate within the Bayesian paradigm, relying on Gaussian processes as our models. These models generate predictive probability density functions (pdfs), and the objective is to integrate them systematically, employing both linear and log-linear pooling. We introduce novel approaches for log-linear pooling, determining input-dependent weights for the predictive pdfs of the Gaussian processes. The aggregation of the pdfs is realized through Monte Carlo sampling, drawing samples of weights from their posterior. The performance of these methods, as well as those based on linear pooling, is demonstrated using a synthetic dataset.
Paper Structure (13 sections, 17 equations, 3 figures)

This paper contains 13 sections, 17 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Gaussian Processes
  4. Random Fourier Feature-based Gaussian Processes
  5. Bayesian Inference via Monte Carlo Algorithms
  6. Fusion of Gaussian processes predictions
  7. Bayesian Hierarchical Stacking
  8. Mixture of GP Experts
  9. Log-Linear Pooling for Stacking and Joint Learning
  10. Experiments
  11. Performance vs number of experts
  12. Performance vs number of spectral frequencies
  13. Conclusions

Figures (3)

  • Figure 1: Bayesian plate diagrams of joint learning and stacking for Gaussian process fusion. The only difference is whether the stochastic function $f_k$ is treated as an input to be conditioned on (as in (a)), or a random variable (as in (b)).
  • Figure 2: A depiction of the generative model and the point predictions of each method. From top to bottom: the means of each component GP and the respective weight process; sample draws from each GP with linewidth indicating $w_k(x)$; the resulting dataset with the predictions of each method.
  • Figure 3: Results from the experiments described in (a) Section \ref{['sec:NLPD_vs_K']} and (b) Section \ref{['sec:NLPD_vs_M']}. The NLPD (lower is better) are averaged across test data; boxes show the median and quartiles of the average across $5$ random data splits for each fusion method, with whiskers showing the minimum and maximum values. In (a), the NLPD of the het-RFF-GP is denoted as a line, as it does not change with $K$.