Entry Dependent Expert Selection in Distributed Gaussian Processes Using Multilabel Classification

TL;DR

This work tackles the scalability of Gaussian Process regression by distributing training across local experts and addresses the high cost of dependency-based aggregation. It replaces static, fixed expert selection with an entry-dependent mechanism founded on multi-label classification to assign test points to multiple GP experts, preserving the asymptotic properties of dependence-aware estimators like NPAE. The authors propose two practical classifiers, KNN and DNN, to perform the expert selection, achieving competitive prediction quality with substantially reduced computation compared to full NPAE and GGM-based methods. Across synthetic and real-world datasets, the classification-based selection yields improved or comparable accuracy and significantly better scalability, making it suitable for distributed and federated learning scenarios.

Abstract

By distributing the training process, local approximation reduces the cost of the standard Gaussian Process. An ensemble technique combines local predictions from Gaussian experts trained on different partitions of the data. Ensemble methods aggregate models' predictions by assuming a perfect diversity of local predictors. Although it keeps the aggregation tractable, this assumption is often violated in practice. Even though ensemble methods provide consistent results by assuming dependencies between experts, they have a high computational cost, which is cubic in the number of experts involved. By implementing an expert selection strategy, the final aggregation step uses fewer experts and is more efficient. However, a selection approach that assigns a fixed set of experts to each new data point cannot encode the specific properties of each unique data point. This paper proposes a flexible expert selection approach based on the characteristics of entry data points. To this end, we investigate the selection task as a multi-label classification problem where the experts define labels, and each entry point is assigned to some experts. The proposed solution's prediction quality, efficiency, and asymptotic properties are discussed in detail. We demonstrate the efficacy of our method through extensive numerical experiments using synthetic and real-world data sets.

Figures (12)

• Figure 1: Computational graphs of an aggregation based on conditional independence assumption between experts.
• Figure 2: Ablation experiment1. Prediction quality of different aggregation baselines for the Concrete data set.
• Figure 3: Expert Selection using GGM for a set of 10 local experts from the Concrete data set: (a) the experts' graph and (b) selected experts based on $60\%$ of most important experts (green nodes).
• Figure 4: Expert Selection scheme of both static and entry-dependent models for a setting of 5 experts with 10 test points. Both selection models assign 3 experts to each test point: (a) original set of experts $\mathcal{E}$, (b) static assignment of experts $\mathcal{E}^\mathcal{G}$, and (c) entry-based selection of experts $\mathcal{E}^\mathcal{C}$.
• Figure 5: Adopted K-nearest neighbors for multi-label classification.

Theorems & Definitions (4)

• Definition 1: Dependency-Based Aggregation
• Example 1: Concrete Data Set
• Definition 2: GGM-related Expert Importance
• Example 2: Expert Selection Models