Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A mixed-categorical correlation kernel for Gaussian process

P. Saves, Y. Diouane, N. Bartoli, T. Lefebvre, J. Morlier

TL;DR

The paper addresses surrogate modeling for expensive mixed input problems by extending exponential kernels to mixed continuous, integer and categorical inputs, forming a GP surrogate with a product-based correlation k(w^r,w^s) = k^{cont}(x^r,x^s) k^{int}(z^r,z^s) k^{cat}(c^r,c^s). It introduces an Exponential Homoscedastic Hypersphere (EHH) kernel family that unifies Gower-like and continuous-relaxation approaches, proves SPD correlation matrices, and demonstrates that EHH generalizes CR and GD while often outperforming FE in analytic and engineering benchmarks. The approach is implemented in SMT v2.0 and evaluated on cosine, cantilever beam, and aircraft design problems, showing improved likelihood and lower residuals with tractable hyperparameter counts. The work points to future enhancements such as KPLS-based dimension reduction to handle larger dimensionality and more complex mixed inputs, enabling efficient Bayesian optimization in industrial settings.

Abstract

Recently, there has been a growing interest for mixed-categorical meta-models based on Gaussian process (GP) surrogates. In this setting, several existing approaches use different strategies either by using continuous kernels (e.g., continuous relaxation and Gower distance based GP) or by using a direct estimation of the correlation matrix. In this paper, we present a kernel-based approach that extends continuous exponential kernels to handle mixed-categorical variables. The proposed kernel leads to a new GP surrogate that generalizes both the continuous relaxation and the Gower distance based GP models. We demonstrate, on both analytical and engineering problems, that our proposed GP model gives a higher likelihood and a smaller residual error than the other kernel-based state-of-the-art models. Our method is available in the open-source software SMT.

A mixed-categorical correlation kernel for Gaussian process

TL;DR

The paper addresses surrogate modeling for expensive mixed input problems by extending exponential kernels to mixed continuous, integer and categorical inputs, forming a GP surrogate with a product-based correlation k(w^r,w^s) = k^{cont}(x^r,x^s) k^{int}(z^r,z^s) k^{cat}(c^r,c^s). It introduces an Exponential Homoscedastic Hypersphere (EHH) kernel family that unifies Gower-like and continuous-relaxation approaches, proves SPD correlation matrices, and demonstrates that EHH generalizes CR and GD while often outperforming FE in analytic and engineering benchmarks. The approach is implemented in SMT v2.0 and evaluated on cosine, cantilever beam, and aircraft design problems, showing improved likelihood and lower residuals with tractable hyperparameter counts. The work points to future enhancements such as KPLS-based dimension reduction to handle larger dimensionality and more complex mixed inputs, enabling efficient Bayesian optimization in industrial settings.

Abstract

Recently, there has been a growing interest for mixed-categorical meta-models based on Gaussian process (GP) surrogates. In this setting, several existing approaches use different strategies either by using continuous kernels (e.g., continuous relaxation and Gower distance based GP) or by using a direct estimation of the correlation matrix. In this paper, we present a kernel-based approach that extends continuous exponential kernels to handle mixed-categorical variables. The proposed kernel leads to a new GP surrogate that generalizes both the continuous relaxation and the Gower distance based GP models. We demonstrate, on both analytical and engineering problems, that our proposed GP model gives a higher likelihood and a smaller residual error than the other kernel-based state-of-the-art models. Our method is available in the open-source software SMT.
Paper Structure (17 sections, 2 theorems, 36 equations, 9 figures, 7 tables)

This paper contains 17 sections, 2 theorems, 36 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. GP for mixed-categorical inputs
  3. Correlation matrices for continuous and integer inputs
  4. Correlation matrices for categorical inputs
  5. Gower distance based kernel
  6. Continuous relaxation based kernel
  7. Homoscedastic hypersphere kernel
  8. An exponential kernel-based model for categorical inputs
  9. Results and discussion
  10. Implementation details
  11. Analytic validation: a categorical cosine problem ($n= 1$, $m=0$, $l=1$ and $L_1=13$)
  12. Application to engineering problems
  13. Cantilever beam bending problem ($n=2$, $m=0$, $l=1$ and $L_1=12$)
  14. Aircraft design application ($n=10$, $m=0$, $l=2$ and $L_1=9$, $L_2=2$)
  15. Conclusion
  16. ...and 2 more sections

Key Result

Theorem 1

Assume that, for all $i \in \{1, \ldots, l\}$, $\Phi(\Theta_i)$ satisfies the parameterization of Eq. (eq:hs_exp). Then the matrix $R^{cat}(\theta^{cat})$, given by Eq. (eq:nat_ext_gaussian_ker:0), is SPD with elements in $[0,1]$.

Figures (9)

  • Figure 1: Mean predictions for our proposed model using different kernels over the matrix $\Theta_1$ for the cosine problem with a 98 point DoE.
  • Figure 2: Mean predictions comparison between EHH and FE kernels over the matrix $\Theta_1$ for the cosine problem with a 98 point DoE.
  • Figure 3: Correlation matrix $R_1^{cat}$ using different choices for $\Theta_1$ on the cosine problem with a DoE of 98 points.
  • Figure 4: RMSE and CPU time to compute models with respect to DoE size.
  • Figure 5: Cantilever beam problem.
  • ...and 4 more figures

Theorems & Definitions (7)

  • Remark 1
  • Remark 2
  • Theorem 1
  • proof
  • Remark 3
  • Theorem 2
  • proof