Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regularized Multi-output Gaussian Convolution Process with Domain Adaptation

Wang Xinming, Wang Chao, Song Xuan, Kirby Levi, Wu Jianguo

TL;DR

This work addresses transfer learning with multi-output Gaussian processes by tackling negative transfer and input-domain mismatch. It introduces a regularized multi-output Gaussian convolution process (MGCP-R) that employs a sparse, non-separable covariance via convolution processes and global regularization to selectively transfer information from sources to a target. A domain adaptation method, DAME, marginalizes extraneous features and expands missing ones to align source inputs with the target, preserving marginal information and enabling effective cross-domain transfer. Theoretical guarantees on consistency and sparsity are provided, and the framework achieves superior performance in simulated studies and a ceramic-density case, while offering computational advantages over full-covariance MGCP models. The approach has practical impact for robust cross-domain transfer learning in manufacturing and other domains where domain mismatch and negative transfer are prevalent, with potential extensions to higher-dimensional data and classification tasks.

Abstract

Multi-output Gaussian process (MGP) has been attracting increasing attention as a transfer learning method to model multiple outputs. Despite its high flexibility and generality, MGP still faces two critical challenges when applied to transfer learning. The first one is negative transfer, which occurs when there exists no shared information among the outputs. The second challenge is the input domain inconsistency, which is commonly studied in transfer learning yet not explored in MGP. In this paper, we propose a regularized MGP modeling framework with domain adaptation to overcome these challenges. More specifically, a sparse covariance matrix of MGP is proposed by using convolution process, where penalization terms are added to adaptively select the most informative outputs for knowledge transfer. To deal with the domain inconsistency, a domain adaptation method is proposed by marginalizing inconsistent features and expanding missing features to align the input domains among different outputs. Statistical properties of the proposed method are provided to guarantee the performance practically and asymptotically. The proposed framework outperforms state-of-the-art benchmarks in comprehensive simulation studies and one real case study of a ceramic manufacturing process. The results demonstrate the effectiveness of our method in dealing with both the negative transfer and the domain inconsistency.

Regularized Multi-output Gaussian Convolution Process with Domain Adaptation

TL;DR

This work addresses transfer learning with multi-output Gaussian processes by tackling negative transfer and input-domain mismatch. It introduces a regularized multi-output Gaussian convolution process (MGCP-R) that employs a sparse, non-separable covariance via convolution processes and global regularization to selectively transfer information from sources to a target. A domain adaptation method, DAME, marginalizes extraneous features and expands missing ones to align source inputs with the target, preserving marginal information and enabling effective cross-domain transfer. Theoretical guarantees on consistency and sparsity are provided, and the framework achieves superior performance in simulated studies and a ceramic-density case, while offering computational advantages over full-covariance MGCP models. The approach has practical impact for robust cross-domain transfer learning in manufacturing and other domains where domain mismatch and negative transfer are prevalent, with potential extensions to higher-dimensional data and classification tasks.

Abstract

Multi-output Gaussian process (MGP) has been attracting increasing attention as a transfer learning method to model multiple outputs. Despite its high flexibility and generality, MGP still faces two critical challenges when applied to transfer learning. The first one is negative transfer, which occurs when there exists no shared information among the outputs. The second challenge is the input domain inconsistency, which is commonly studied in transfer learning yet not explored in MGP. In this paper, we propose a regularized MGP modeling framework with domain adaptation to overcome these challenges. More specifically, a sparse covariance matrix of MGP is proposed by using convolution process, where penalization terms are added to adaptively select the most informative outputs for knowledge transfer. To deal with the domain inconsistency, a domain adaptation method is proposed by marginalizing inconsistent features and expanding missing features to align the input domains among different outputs. Statistical properties of the proposed method are provided to guarantee the performance practically and asymptotically. The proposed framework outperforms state-of-the-art benchmarks in comprehensive simulation studies and one real case study of a ceramic manufacturing process. The results demonstrate the effectiveness of our method in dealing with both the negative transfer and the domain inconsistency.
Paper Structure (28 sections, 3 theorems, 63 equations, 12 figures, 6 tables, 1 algorithm)

This paper contains 28 sections, 3 theorems, 63 equations, 12 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Introduction
  3. Preliminaries
  4. Related works on MGCP
  5. Multi-output Gaussian Process
  6. Convolution Process
  7. Model development
  8. Regularized MGCP modeling framework
  9. Statistical properties for regularized MGCP
  10. Domain adaptation through marginalization and expansion (DAME)
  11. Implemetation using Gaussian kernel and $L_1$ norm
  12. Unique Methodology Contribution
  13. Numerical studies
  14. General settings
  15. Simulation case I
  16. ...and 13 more sections

Key Result

Theorem 1

Suppose that $g_{it}(\bm{x})=0, \forall i \in \mathcal{U} \subseteq \mathcal{I}^S$ for all $\bm{x}\in \mathcal{X}$. For notational convenience, suppose $\mathcal{U}=\{1,2,...,h|h\leq q\}$, then the predictive distribution of the model at any new input $\bm{x}_*$ is unrelated with $\{f_1,f_2,...,f_h where $\bm{k}_{+}=(\bm{K}_{h+1,*}^T,...,\bm{K}_{q,*}^T,\bm{K}_{t,*}^T)^T$, $\bm{y}_{+}=(\bm{y}_{h+1

Figures (12)

  • Figure 1: Graphical model of convolution process, where $\ast$ denotes a convolution operation.
  • Figure 2: The structure of MGP Kontar2018 for modeling the target output $f_t$.
  • Figure 3: Illustration of the marginalization based domain adaptation using the normalized density data of ceramic product. (a). Marginalize source data to the domain only with feature $x^{(c)}$, and obtain marginal distribution through kernel regression; (b). Induce data based on the marginal distribution and expand them to the target domain.
  • Figure 4: Boxplots of the MAE in the simulation case I, where the line in each box represents the median value.
  • Figure 5: Visualization of results in one repetition of 1D example.
  • ...and 7 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2
  • Theorem 3