Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Low-Dimensional Embeddings for Black-Box Optimization

Riccardo Busetto, Manas Mejari, Marco Forgione, Alberto Bemporad, Dario Piga

TL;DR

The paper tackles the scalability of black-box optimization in high dimensions by learning a low-dimensional latent manifold of optimal solutions for a problem class parameterized by \(\theta\). It introduces an autoencoder-based embedding learned from a meta-dataset of near-optimal solutions, enabling GLIS-based optimization in the latent space (Meta-GLIS). A distribution-free probabilistic bound on the sub-optimality of latent-space optimization is derived, and the approach is demonstrated on a Rosenbrock benchmark and MPC hyperparameter calibration, showing faster convergence and reduced evaluations compared to full-space GLIS. The method offers practical impact by reducing experimental or simulation cost when tuning complex systems, with interpretable variants via a linear decoder and avenues for future theoretical and methodological refinements.

Abstract

When gradient-based methods are impractical, black-box optimization (BBO) provides a valuable alternative. However, BBO often struggles with high-dimensional problems and limited trial budgets. In this work, we propose a novel approach based on meta-learning to pre-compute a reduced-dimensional manifold where optimal points lie for a specific class of optimization problems. When optimizing a new problem instance sampled from the class, black-box optimization is carried out in the reduced-dimensional space, effectively reducing the effort required for finding near-optimal solutions.

Learning Low-Dimensional Embeddings for Black-Box Optimization

TL;DR

The paper tackles the scalability of black-box optimization in high dimensions by learning a low-dimensional latent manifold of optimal solutions for a problem class parameterized by . It introduces an autoencoder-based embedding learned from a meta-dataset of near-optimal solutions, enabling GLIS-based optimization in the latent space (Meta-GLIS). A distribution-free probabilistic bound on the sub-optimality of latent-space optimization is derived, and the approach is demonstrated on a Rosenbrock benchmark and MPC hyperparameter calibration, showing faster convergence and reduced evaluations compared to full-space GLIS. The method offers practical impact by reducing experimental or simulation cost when tuning complex systems, with interpretable variants via a linear decoder and avenues for future theoretical and methodological refinements.

Abstract

When gradient-based methods are impractical, black-box optimization (BBO) provides a valuable alternative. However, BBO often struggles with high-dimensional problems and limited trial budgets. In this work, we propose a novel approach based on meta-learning to pre-compute a reduced-dimensional manifold where optimal points lie for a specific class of optimization problems. When optimizing a new problem instance sampled from the class, black-box optimization is carried out in the reduced-dimensional space, effectively reducing the effort required for finding near-optimal solutions.
Paper Structure (14 sections, 2 theorems, 38 equations, 8 figures, 1 algorithm)

This paper contains 14 sections, 2 theorems, 38 equations, 8 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem description
  3. Methodology
  4. Meta-dataset creation
  5. Learning a latent representation via autoencoders
  6. Meta-GLIS: GLIS over a low-dimensional space
  7. Sub-optimality quantification
  8. Definitions
  9. Generalization bounds on the performance gap
  10. Numerical examples
  11. Rosenbrock function benchmark
  12. Automated calibration of MPC hyperparameters
  13. Linear Decoder
  14. Conclusions

Key Result

Theorem 1

For any $\alpha,\delta \in (0,1)$, let $k^\star := \lceil m\,(1-\alpha+\varepsilon_m)\rceil$ and $\varepsilon_m := \sqrt{\tfrac{\log(2/\delta)}{2m}}$. Assume that the number of validation samples $m$ is large enough so that $\alpha \geq \varepsilon_m$ (or equivalently, $k^\star \le m$). Then, with where $\Psi_{(k)}$ denotes the $k$-th order statistic of the validation values, i.e., if then $\Ps

Figures (8)

  • Figure 1: Learned manifold of the trained autoencoder over test functions: candidate solutions generated by the differential evolution algorithm (blue points); optimal solution for each problem instance (green points); output of the autoencoder for each candidate solution (orange points).
  • Figure 2: Comparison of the average performance obtained with GLIS (blue) and Meta-GLIS (red) over $N^{\mathrm{test}} = 100$ runs. Global optimum (green). Shaded areas correspond $\pm$ one standard deviation over $100$ Monte Carlo runs.
  • Figure 3: Sampled points by GLIS (blue circles) and Meta-GLIS (orange circles), along with corresponding best solutions (stars). The global optimal solution (green cross) overlaps with the orange star.
  • Figure 4: Empirical CDFs of the performance gap using Meta-GLIS compared with PSO (panel (a)) and GLIS (panel (b)) over $100$ test instances. Theoretical bound from Theorem \ref{['theorem:gap']} (blue line); empirical 90%-quantile on the test set (orange line).
  • Figure 5: Empirical distributions of the optimal solutions over $100$ test system realizations (solid lines) and corresponding $3000$ queries sampled during the optimization (shaded areas) for each MPC hyperparameter: GLIS (blue); Meta-GLIS with nonlinear (orange) and linear decoder (purple).
  • ...and 3 more figures

Theorems & Definitions (4)

  • Remark 1
  • Remark 2
  • Theorem 1: Generalization bounds
  • Lemma 1: DKW--Massart inequality massart1990tight