Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decentralized Online Ensembles of Gaussian Processes for Multi-Agent Systems

Fernando Llorente, Daniel Waxman, Petar M. Djurić

TL;DR

The paper tackles decentralized Bayesian regression for multi-agent systems by proposing Random Feature Gaussian Processes (RF-GP) to enable online, distributed inference without a central fusion center. It derives an exact Bayesian update in RF-GP form and shows how local, additive information accumulates to a global posterior via consensus, achieving asymptotic agreement. To handle hyperparameters, it introduces online Bayesian model averaging (BMA) across an ensemble of RF-GP models, with decentralized schemes to compute model weights. Empirical results on real-world datasets demonstrate that the decentralized approach can rival centralized SVI-GP performance while offering scalability, making it suitable for large-scale, distributed multi-agent applications.

Abstract

Flexible and scalable decentralized learning solutions are fundamentally important in the application of multi-agent systems. While several recent approaches introduce (ensembles of) kernel machines in the distributed setting, Bayesian solutions are much more limited. We introduce a fully decentralized, asymptotically exact solution to computing the random feature approximation of Gaussian processes. We further address the choice of hyperparameters by introducing an ensembling scheme for Bayesian multiple kernel learning based on online Bayesian model averaging. The resulting algorithm is tested against Bayesian and frequentist methods on simulated and real-world datasets.

Decentralized Online Ensembles of Gaussian Processes for Multi-Agent Systems

TL;DR

The paper tackles decentralized Bayesian regression for multi-agent systems by proposing Random Feature Gaussian Processes (RF-GP) to enable online, distributed inference without a central fusion center. It derives an exact Bayesian update in RF-GP form and shows how local, additive information accumulates to a global posterior via consensus, achieving asymptotic agreement. To handle hyperparameters, it introduces online Bayesian model averaging (BMA) across an ensemble of RF-GP models, with decentralized schemes to compute model weights. Empirical results on real-world datasets demonstrate that the decentralized approach can rival centralized SVI-GP performance while offering scalability, making it suitable for large-scale, distributed multi-agent applications.

Abstract

Flexible and scalable decentralized learning solutions are fundamentally important in the application of multi-agent systems. While several recent approaches introduce (ensembles of) kernel machines in the distributed setting, Bayesian solutions are much more limited. We introduce a fully decentralized, asymptotically exact solution to computing the random feature approximation of Gaussian processes. We further address the choice of hyperparameters by introducing an ensembling scheme for Bayesian multiple kernel learning based on online Bayesian model averaging. The resulting algorithm is tested against Bayesian and frequentist methods on simulated and real-world datasets.

Paper Structure

This paper contains 14 sections, 1 theorem, 23 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Random Feature Gaussian Processes
  4. Gaussian Processes
  5. Random Feature Gaussian Processes
  6. Distributed Bayesian Inference for RF-GPs
  7. Distributed Computation with Fusion Center
  8. A Fusion Perspective
  9. Decentralized Bayesian Inference
  10. An Ensembling Approach
  11. Bayesian Model Averaging
  12. The Issue in the Decentralized Setting and a Solution
  13. Experiments and Discussion
  14. Conclusion

Key Result

Lemma 1

Let $p_1(\boldsymbol{\mathbf{z}}) = \mathcal{N}(\mu_1, \Sigma_1)$ and $p_2(\boldsymbol{\mathbf{z}}) = \mathcal{N}(\mu_2, \Sigma_2)$ be two Gaussian densities on the quantity $\boldsymbol{\mathbf{z}}$. Then the normalized density $p_3(\boldsymbol{\mathbf{z}}) \propto p_1(\boldsymbol{\mathbf{z}}) \tim

Figures (1)

  • Figure 1: The comparative performance of D-RF-GP, DOMKL, AdaRaker, and an SVI GP.

Theorems & Definitions (1)

  • Lemma 1