Learning Multi-Task Gaussian Process Over Heterogeneous Input Domains

TL;DR

A novel heterogeneous stochastic variational linear model of coregionalization (HSVLMC) model for simultaneously learning the tasks with varied input domains and employs a residual modeling strategy to leverage the inductive bias brought by prior domain mappings for better-model inference.

Abstract

Multi-task Gaussian process (MTGP) is a well-known non-parametric Bayesian model for learning correlated tasks effectively by transferring knowledge across tasks. But current MTGPs are usually limited to the multi-task scenario defined in the same input domain, leaving no space for tackling the heterogeneous case, i.e., the features of input domains vary over tasks. To this end, this paper presents a novel heterogeneous stochastic variational linear model of coregionalization (HSVLMC) model for simultaneously learning the tasks with varied input domains. Particularly, we develop the stochastic variational framework with Bayesian calibration that (i) takes into account the effect of dimensionality reduction raised by domain mappings in order to achieve effective input alignment; and (ii) employs a residual modeling strategy to leverage the inductive bias brought by prior domain mappings for better model inference. Finally, the superiority of the proposed model against existing LMC models has been extensively verified on diverse heterogeneous multi-task cases and a practical multi-fidelity steam turbine exhaust problem.

Figures (9)

• Figure 1: Illustration of the predictions of HSVLMC on the noisy case, with the shaded region representing 95% confidence interval. Note that the crosses are training data, the dot vertical lines indicate the mean $\mu_g^1$ of aligned inputs via the GP-inspired domain mapping $g^1(.)$, while the solid vertical lines represent the mean $\mu_{g_0}^1$ of aligned inputs via the prior domain mapping $g_0^1(.)$.
• Figure 2: Boxplots of different GP models on the multi-fidelity case in terms of the SMSE and SMLL criteria.
• Figure 3: Predictions versus observations and the prior/posterior aligned inputs of different LMC models on the multi-fidelity case. Note that the crosses in the upper plots represent the prediction mean, while the error bars indicate 95% confidence interval; the red circles in the bottom plots represent the means $\{\bm{\mu}_{g,i}^1\}$ of aligned inputs via the GP-inspired domain mapping $g^1(.)$, while the triangles represent the means $\{\bm{\mu}_{g_0,i}^1\}$ of aligned inputs via the prior domain mapping $g_0^1(.)$.
• Figure 4: The violin plot of prior variance $\nu_{g_0}^1$ learned by the proposed HSVLMC on the multi-fidelity case over ten runs.
• Figure 5: Impact of training size $N^1$ for the target task on the performance of heterogeneous LMC models on the $\mathtt{airfoil}$ case.