GGMPs: Generalized Gaussian Mixture Processes

TL;DR

The Generalized Gaussian Mixture Process (GGMP), a GP-based method for multimodal conditional density estimation in settings where each input may be associated with a complex output distribution rather than a single scalar response, is introduced.

Abstract

Conditional density estimation is complicated by multimodality, heteroscedasticity, and strong non-Gaussianity. Gaussian processes (GPs) provide a principled nonparametric framework with calibrated uncertainty, but standard GP regression is limited by its unimodal Gaussian predictive form. We introduce the Generalized Gaussian Mixture Process (GGMP), a GP-based method for multimodal conditional density estimation in settings where each input may be associated with a complex output distribution rather than a single scalar response. GGMP combines local Gaussian mixture fitting, cross-input component alignment and per-component heteroscedastic GP training to produce a closed-form Gaussian mixture predictive density. The method is tractable, compatible with standard GP solvers and scalable methods, and avoids the exponentially large latent-assignment structure of naive multimodal GP formulations. Empirically, GGMPs improve distributional approximation on synthetic and real-world datasets with pronounced non-Gaussianity and multimodality.

Figures (4)

• Figure 1: Generalized Gaussian Mixture Processes (GGMPs) capture multimodal conditional distributions while preserving closed-form inference. Many real-world processes exhibit conditioning-dependent multimodality, asymmetry, and heteroscedasticity that standard GPs cannot represent. GGMPs address this by modeling $p(y \mid x)$ as a weighted mixture of GPs. Here, predictive densities at four held-out inputs are shown against empirical conditional distributions (blue histograms). With the number of components in the mixture $K{=}1$ (orange), GGMPs reduce to a standard heteroscedastic GP. Increasing $K$ progressively resolves multimodal structure, with the model remaining robust to overspecification as surplus components overlap with existing ones.
• Figure 2: Synthetic distribution field with fixed-point slices. The background shows $\log_{10} p(y\mid x)$, generated from a single latent mean function $f(x)$ (black curve), while red vertical slices depict sampled local conditional densities at selected $x$-locations. Despite sharing the same underlying $f(x)$, the local distributions vary strongly in modality, spread, and asymmetry.
• Figure 3: Joint density reconstruction on held-out conditions (multivariate, $K=5$). Blue contours show the empirical test distribution in the 2D output space (Axis 1–Axis 2), and red contours show the GGMP$_5$ predictive distribution at the same inputs. Each panel corresponds to a different held-out condition. Overall alignment of contour shape and multimodal structure indicates that the GGMP captures the major geometry of the joint distribution.
• Figure 4: Held-out distribution reconstruction across mixture complexity for the synthetic function. Each row shows a different number of mixture components; each column is one of four approximately equally spaced held-out inputs. In every panel, the ground-truth is shown in gray, and the model prediction is red. Increasing $K$ improves local shape fidelity—especially multimodal structure—while low $K$ underfits fine-scale features. When $K < K_{\text{true}}$, we witness an averaging phenomenon; and when $K > K_{\text{true}}$, we observe a distribution that looks like it could have come from a mixture with fewer components, indicating that components are likely stacked.

Theorems & Definitions (3)

• Proposition 3.1: Universal conditional density approximation
• Lemma 3.2: Distributional MLE = KL minimization
• proof