Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiBBO: HiPPO-based Space Consistency for High-dimensional Bayesian Optimisation

Junyu Xuan, Wenlong Chen, Yingzhen Li

TL;DR

HiBBO tackles high-dimensional Bayesian Optimisation by mitigating distribution mismatch between latent and original spaces in VAE-based BO. It injects a HiPPO-based memory regulariser into VAE training to preserve kernel-distance relationships, ensuring the latent GP better reflects the true objective in the original space. The method trains the VAE with a loss that includes a HiPPO consistency term and performs BO in the latent space, achieving faster convergence and higher-quality solutions than strong VAE-BO baselines across Ackley, MNIST-derived, shape, and molecular design tasks. This work links sequence representation with efficient high-dimensional BO and holds promise for scalable neural architecture search and materials/drug design applications.

Abstract

Bayesian Optimisation (BO) is a powerful tool for optimising expensive blackbox functions but its effectiveness diminishes in highdimensional spaces due to sparse data and poor surrogate model scalability While Variational Autoencoder (VAE) based approaches address this by learning low-dimensional latent representations the reconstructionbased objective function often brings the functional distribution mismatch between the latent space and original space leading to suboptimal optimisation performance In this paper we first analyse the reason why reconstructiononly loss may lead to distribution mismatch and then propose HiBBO a novel BO framework that introduces the space consistency into the latent space construction in VAE using HiPPO - a method for longterm sequence modelling - to reduce the functional distribution mismatch between the latent space and original space Experiments on highdimensional benchmark tasks demonstrate that HiBBO outperforms existing VAEBO methods in convergence speed and solution quality Our work bridges the gap between high-dimensional sequence representation learning and efficient Bayesian Optimisation enabling broader applications in neural architecture search materials science and beyond.

HiBBO: HiPPO-based Space Consistency for High-dimensional Bayesian Optimisation

TL;DR

HiBBO tackles high-dimensional Bayesian Optimisation by mitigating distribution mismatch between latent and original spaces in VAE-based BO. It injects a HiPPO-based memory regulariser into VAE training to preserve kernel-distance relationships, ensuring the latent GP better reflects the true objective in the original space. The method trains the VAE with a loss that includes a HiPPO consistency term and performs BO in the latent space, achieving faster convergence and higher-quality solutions than strong VAE-BO baselines across Ackley, MNIST-derived, shape, and molecular design tasks. This work links sequence representation with efficient high-dimensional BO and holds promise for scalable neural architecture search and materials/drug design applications.

Abstract

Bayesian Optimisation (BO) is a powerful tool for optimising expensive blackbox functions but its effectiveness diminishes in highdimensional spaces due to sparse data and poor surrogate model scalability While Variational Autoencoder (VAE) based approaches address this by learning low-dimensional latent representations the reconstructionbased objective function often brings the functional distribution mismatch between the latent space and original space leading to suboptimal optimisation performance In this paper we first analyse the reason why reconstructiononly loss may lead to distribution mismatch and then propose HiBBO a novel BO framework that introduces the space consistency into the latent space construction in VAE using HiPPO - a method for longterm sequence modelling - to reduce the functional distribution mismatch between the latent space and original space Experiments on highdimensional benchmark tasks demonstrate that HiBBO outperforms existing VAEBO methods in convergence speed and solution quality Our work bridges the gap between high-dimensional sequence representation learning and efficient Bayesian Optimisation enabling broader applications in neural architecture search materials science and beyond.

Paper Structure

This paper contains 18 sections, 1 theorem, 12 equations, 6 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background: VAE-based BO
  3. Our Method
  4. Distribution mismatch between reconstructed and original space
  5. Reduce kernel distance via HiPPO memory representation
  6. BO with HiPPO-based space consistency
  7. Related Works
  8. Experiments
  9. Standard function optimisation
  10. MNIST-based synthetic problem
  11. Shape optimisation
  12. Chemical design
  13. Conclusion
  14. Appendix
  15. Notation Table
  16. ...and 3 more sections

Key Result

Proposition 1

If the HiPPO order is larger than the degree of the kernel function, then the closeness of HiPPO representations of two sequences of data would imply their closeness of kernel distance.

Figures (6)

  • Figure 1: The illustration of our idea that is to increase the space consistency between the latent and original spaces by reducing the HiPPO representations of reconstructed data sequence and original data sequence.
  • Figure 2: Empirical demonstration of the capability of HiPPO representation in expressing the correlation between data points.
  • Figure 3: Visualisation of our method.
  • Figure 4: results on standard function optimisation and MNIST-based synthetic problem
  • Figure 5: results on shape optimisation and chemical design
  • ...and 1 more figures

Theorems & Definitions (2)

  • Proposition 1
  • proof