Instance Generation for Meta-Black-Box Optimization through Latent Space Reverse Engineering

TL;DR

This work addresses the limited diversity of training problems in meta-learning for Black-Box Optimization (MetaBBO) and its impact on generalization. It introduces Latent Space Reverse Engineering (LSRE), which maps Exploratory Landscape Analysis features into a 2D latent space via a neural autoencoder and then uses Genetic Programming to reverse-engineer diverse problem formulas that populate a Diverse-BBO benchmark. Through grid sampling in the latent space and distributed GP-based search, LSRE yields 256 diversified problem instances, enabling better generalization for MetaBBO approaches across synthetic and realistic scenarios. Ablation studies and extensive experiments demonstrate that Diverse-BBO, produced by LSRE, offers superior diversity and improves cross-domain generalization, highlighting the importance of training-data design in MetaBBO research.

Abstract

To relieve intensive human-expertise required to design optimization algorithms, recent Meta-Black-Box Optimization (MetaBBO) researches leverage generalization strength of meta-learning to train neural network-based algorithm design policies over a predefined training problem set, which automates the adaptability of the low-level optimizers on unseen problem instances. Currently, a common training problem set choice in existing MetaBBOs is well-known benchmark suites CoCo-BBOB. Although such choice facilitates the MetaBBO's development, problem instances in CoCo-BBOB are more or less limited in diversity, raising the risk of overfitting of MetaBBOs, which might further results in poor generalization. In this paper, we propose an instance generation approach, termed as \textbf{LSRE}, which could generate diverse training problem instances for MetaBBOs to learn more generalizable policies. LSRE first trains an autoencoder which maps high-dimensional problem features into a 2-dimensional latent space. Uniform-grid sampling in this latent space leads to hidden representations of problem instances with sufficient diversity. By leveraging a genetic-programming approach to search function formulas with minimal L2-distance to these hidden representations, LSRE reverse engineers a diversified problem set, termed as \textbf{Diverse-BBO}. We validate the effectiveness of LSRE by training various MetaBBOs on Diverse-BBO and observe their generalization performances on either synthetic or realistic scenarios. Extensive experimental results underscore the superiority of Diverse-BBO to existing training set choices in MetaBBOs. Further ablation studies not only demonstrate the effectiveness of design choices in LSRE, but also reveal interesting insights on instance diversity and MetaBBO's generalization.

Figures (7)

• Figure 1: General workflow of MetaBBO approaches.
• Figure 2: Workflow of Latent Instance Space Analysis.
• Figure 3: A proximal symbolic tree example.
• Figure 4: Average Success Rate Distribution
• Figure 5: Generalization performance on realistic problems.