Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weighted Euclidean Distance Matrices over Mixed Continuous and Categorical Inputs for Gaussian Process Models

Mingyu Pu, Songhao Wang, Haowei Wang, Szu Hui Ng

TL;DR

This work introduces WEighed Euclidean Distance Matrices GP (WEGP) to extend Gaussian Processes to mixed continuous and categorical inputs by learning a distance-based kernel for each categorical variable. Each categorical kernel is built as a positive linear combination of base EDMs $D_k=\sum_i w_k^{(i)}D_k^{(i)}$, with weights inferred in a fully Bayesian framework using NUTS, and two base-EDM constructions are proposed: ordinal encodings and extreme directions within the EDM cone. Theoretical results establish convergence of the posterior mean under a Matérn kernel, and empirical studies show state-of-the-art performance in Bayesian Optimization on synthetic and real-world problems, with improved robustness in data-sparse regimes. The approach provides interpretable distance patterns and scalable handling of multi-category factors, offering practical gains for expensive black-box optimization across engineering and scientific domains.

Abstract

Gaussian Process (GP) models are widely utilized as surrogate models in scientific and engineering fields. However, standard GP models are limited to continuous variables due to the difficulties in establishing correlation structures for categorical variables. To overcome this limitati on, we introduce WEighted Euclidean distance matrices Gaussian Process (WEGP). WEGP constructs the kernel function for each categorical input by estimating the Euclidean distance matrix (EDM) among all categorical choices of this input. The EDM is represented as a linear combination of several predefined base EDMs, each scaled by a positive weight. The weights, along with other kernel hyperparameters, are inferred using a fully Bayesian framework. We analyze the predictive performance of WEGP theoretically. Numerical experiments validate the accuracy of our GP model, and by WEGP, into Bayesian Optimization (BO), we achieve superior performance on both synthetic and real-world optimization problems.

Weighted Euclidean Distance Matrices over Mixed Continuous and Categorical Inputs for Gaussian Process Models

TL;DR

This work introduces WEighed Euclidean Distance Matrices GP (WEGP) to extend Gaussian Processes to mixed continuous and categorical inputs by learning a distance-based kernel for each categorical variable. Each categorical kernel is built as a positive linear combination of base EDMs , with weights inferred in a fully Bayesian framework using NUTS, and two base-EDM constructions are proposed: ordinal encodings and extreme directions within the EDM cone. Theoretical results establish convergence of the posterior mean under a Matérn kernel, and empirical studies show state-of-the-art performance in Bayesian Optimization on synthetic and real-world problems, with improved robustness in data-sparse regimes. The approach provides interpretable distance patterns and scalable handling of multi-category factors, offering practical gains for expensive black-box optimization across engineering and scientific domains.

Abstract

Gaussian Process (GP) models are widely utilized as surrogate models in scientific and engineering fields. However, standard GP models are limited to continuous variables due to the difficulties in establishing correlation structures for categorical variables. To overcome this limitati on, we introduce WEighted Euclidean distance matrices Gaussian Process (WEGP). WEGP constructs the kernel function for each categorical input by estimating the Euclidean distance matrix (EDM) among all categorical choices of this input. The EDM is represented as a linear combination of several predefined base EDMs, each scaled by a positive weight. The weights, along with other kernel hyperparameters, are inferred using a fully Bayesian framework. We analyze the predictive performance of WEGP theoretically. Numerical experiments validate the accuracy of our GP model, and by WEGP, into Bayesian Optimization (BO), we achieve superior performance on both synthetic and real-world optimization problems.

Paper Structure

This paper contains 25 sections, 5 theorems, 30 equations, 3 figures, 6 tables, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. RELATED WORK
  4. WEGP
  5. Direct Euclidean Distance Matrix Estimation
  6. Kernel Hyperparameters Estimation
  7. Theoretical Analysis
  8. WEBO
  9. EXPERIMENTS
  10. WEGP
  11. WEBO
  12. DISCUSSION
  13. Theoretical analysis of EDM
  14. Proof of proposition \ref{['proposition:linearindependent']}
  15. Proof of proposition \ref{['prop:extreme']}
  16. ...and 10 more sections

Key Result

Proposition 1

Denote $m$ linearly independent $n \times n$ base EDMs as $D^{(1)}, D^{(2)}, \ldots, D^{(m)}$. Define D as positive linear combination of the matrices $D^{(1)}, D^{(2)}, \ldots, D^{(m)}$: $D$ is also a valid Euclidean distance matrix.

Figures (3)

  • Figure 1: WEGP model accuracy comparison under three different sizes of training data
  • Figure 2: Results on mixed input optimization problems. Lines and shaded area denote mean and standard error.
  • Figure 3: WEGP model accuracy comparison by log likelihood by three different sizes of training data

Theorems & Definitions (16)

  • Definition 1
  • Proposition 1
  • proof
  • Example 1
  • Definition 2: boyd2004convex
  • Remark 1
  • Lemma 1: Carathéodory's theorem
  • Definition 3: Extreme Directions of the EDM Cone
  • Proposition 2
  • proof
  • ...and 6 more