CoCoAFusE: Beyond Mixtures of Experts via Model Fusion

TL;DR

CoCoAFusE addresses the challenge of uncertainty quantification and interpretability in regressing data governed by multiple generating mechanisms. It extends finite Mixtures of Experts by introducing a fusion mechanism controlled by a behavior gate that blends and mixes expert densities, enabling smooth transitions between regimes while preserving local interpretability. The framework is evaluated on synthetic benchmarks and real datasets, showing tighter credible bounds and robust density estimation compared to classical MoEs and BoEs, with competitive performance against Gaussian process regression and Bayesian neural networks. The approach offers practical benefits for safety-critical applications by delivering calibrated predictive densities and explicit modeling of regime interplay, albeit at higher computational cost which motivates future scalable inference developments.

Abstract

Many learning problems involve multiple patterns and varying degrees of uncertainty dependent on the covariates. Advances in Deep Learning (DL) have addressed these issues by learning highly nonlinear input-output dependencies. However, model interpretability and Uncertainty Quantification (UQ) have often straggled behind. In this context, we introduce the Competitive/Collaborative Fusion of Experts (CoCoAFusE), a novel, Bayesian Covariates-Dependent Modeling technique. CoCoAFusE builds on the very philosophy behind Mixtures of Experts (MoEs), blending predictions from several simple sub-models (or "experts") to achieve high levels of expressiveness while retaining a substantial degree of local interpretability. Our formulation extends that of a classical Mixture of Experts by contemplating the fusion of the experts' distributions in addition to their more usual mixing (i.e., superimposition). Through this additional feature, CoCoAFusE better accommodates different scenarios for the intermediate behavior between generating mechanisms, resulting in tighter credible bounds on the response variable. Indeed, only resorting to mixing, as in classical MoEs, may lead to multimodality artifacts, especially over smooth transitions. Instead, CoCoAFusE can avoid these artifacts even under the same structure and priors for the experts, leading to greater expressiveness and flexibility in modeling. This new approach is showcased extensively on a suite of motivating numerical examples and a collection of real-data ones, demonstrating its efficacy in tackling complex regression problems where uncertainty is a key quantity of interest.

Figures (10)

• Figure 1: The Bayesian research cycle (adapted from 2021vandeschootBayesianStatisticsModellinga). a. The standard research cycle. b. The Bayesian-specific workflow, involving prior elicitation, choice of likelihood, obtaining the posterior, and performing inference.
• Figure 2: Two interpolation approaches between the mixture (forefront) and blend density (back). Left: the probability distribution for each $\beta$ is a naïve interpolation of mixture and blend, resulting in intermediate densities with an additional mode. Right: the parameters of the base densities are interpolated between the extreme cases as a function of $\beta$ and then mixed, preserving the number of modes.
• Figure 3: Schematic overview of the MoE, BoE, and the CoCoAFusE. Blocks of the same color are identical in the two models, with dotted lines representing the sharing of parameters. In the MoE, multiple distributions are mixed via a gate (pink dashed rectangle). In the BoE, the experts are first blended, and then predictions are generated (aqua dashed rectangle). In CoCoAFusE, we "fuse" experts -- mix different blends of their densities --, as a function of the behavior gate $\beta$'s output.
• Figure 4: Plate Diagrams for the MoE and CoCoAFusE. Blue shaded circles represent observed data, light pink shaded circles indicate uncertain parameters, sand rectangles are prior hyperparameters, and plates represent sub-graphs that are repeated (plate $M$ corresponds to the various experts, and plate $N$ corresponds to data points). Arrows represent conditional dependence.
• Figure 5: Predictions for MoE, BoE, and CoCoAFusE on the Switch Example.

Theorems & Definitions (30)

• Theorem 1: Bayes' Theorem
• Remark 1: Prior, Likelihood, and Posterior Distributions
• Remark 2: Noteworthy Examples
• Remark 3: Interpolation of Variances
• Lemma 1
• Lemma 2
• Remark 4: Naïve Interpolation between Densities
• Example 1
• Lemma 3
• Corollary 1