Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMOG: Scalable Meta-Learning for Multi-Objective Bayesian Optimization

Leonard Papenmeier, Petru Tighineanu

TL;DR

SMOG introduces a scalable meta-learning framework for multi-objective Bayesian optimization by learning correlations across $M$ meta-tasks and a target task with $O$ objectives via a modular multi-output GP. The method imposes two sparsity-driven assumptions to yield a tractable prior where the target is an additive combination of meta-task posteriors plus a residual, with weights $w_{mo}$ learned by marginal likelihood, ensuring principled uncertainty propagation. Complexity scales linearly with the number of meta-tasks, enabling training with many tasks and caching meta-task posteriors for efficient target learning. Empirical results on Sinusoidal, Adapted Hartmann6, HPOBench, and Terrain benchmarks show SMOG provides fast initial Pareto-front discovery and robust performance by leveraging cross-task correlations. The work offers a practical, uncertainty-aware mediator between meta-learning and MOBO that can accelerate real-world multi-objective optimization problems.

Abstract

Multi-objective optimization aims to solve problems with competing objectives, often with only black-box access to a problem and a limited budget of measurements. In many applications, historical data from related optimization tasks is available, creating an opportunity for meta-learning to accelerate the optimization. Bayesian optimization, as a promising technique for black-box optimization, has been extended to meta-learning and multi-objective optimization independently, but methods that simultaneously address both settings - meta-learned priors for multi-objective Bayesian optimization - remain largely unexplored. We propose SMOG, a scalable and modular meta-learning model based on a multi-output Gaussian process that explicitly learns correlations between objectives. SMOG builds a structured joint Gaussian process prior across meta- and target tasks and, after conditioning on metadata, yields a closed-form target-task prior augmented by a flexible residual multi-output kernel. This construction propagates metadata uncertainty into the target surrogate in a principled way. SMOG supports hierarchical, parallel training: meta-task Gaussian processes are fit once and then cached, achieving linear scaling with the number of meta-tasks. The resulting surrogate integrates seamlessly with standard multi-objective Bayesian optimization acquisition functions.

SMOG: Scalable Meta-Learning for Multi-Objective Bayesian Optimization

TL;DR

SMOG introduces a scalable meta-learning framework for multi-objective Bayesian optimization by learning correlations across meta-tasks and a target task with objectives via a modular multi-output GP. The method imposes two sparsity-driven assumptions to yield a tractable prior where the target is an additive combination of meta-task posteriors plus a residual, with weights learned by marginal likelihood, ensuring principled uncertainty propagation. Complexity scales linearly with the number of meta-tasks, enabling training with many tasks and caching meta-task posteriors for efficient target learning. Empirical results on Sinusoidal, Adapted Hartmann6, HPOBench, and Terrain benchmarks show SMOG provides fast initial Pareto-front discovery and robust performance by leveraging cross-task correlations. The work offers a practical, uncertainty-aware mediator between meta-learning and MOBO that can accelerate real-world multi-objective optimization problems.

Abstract

Multi-objective optimization aims to solve problems with competing objectives, often with only black-box access to a problem and a limited budget of measurements. In many applications, historical data from related optimization tasks is available, creating an opportunity for meta-learning to accelerate the optimization. Bayesian optimization, as a promising technique for black-box optimization, has been extended to meta-learning and multi-objective optimization independently, but methods that simultaneously address both settings - meta-learned priors for multi-objective Bayesian optimization - remain largely unexplored. We propose SMOG, a scalable and modular meta-learning model based on a multi-output Gaussian process that explicitly learns correlations between objectives. SMOG builds a structured joint Gaussian process prior across meta- and target tasks and, after conditioning on metadata, yields a closed-form target-task prior augmented by a flexible residual multi-output kernel. This construction propagates metadata uncertainty into the target surrogate in a principled way. SMOG supports hierarchical, parallel training: meta-task Gaussian processes are fit once and then cached, achieving linear scaling with the number of meta-tasks. The resulting surrogate integrates seamlessly with standard multi-objective Bayesian optimization acquisition functions.
Paper Structure (40 sections, 6 theorems, 36 equations, 15 figures, 1 algorithm)

This paper contains 40 sections, 6 theorems, 36 equations, 15 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contribution.
  3. Related Work
  4. Method
  5. Method Description
  6. Example for two meta-tasks and objectives.
  7. Likelihood optimization
  8. Complexity Analysis
  9. Experimental Setup
  10. Equicorrelated Multitask Kernel
  11. Benchmarks
  12. Sinusoidal Benchmark.
  13. Adapted Hartmann6 Benchmark.
  14. Tabular HPO Benchmarks
  15. Terrain Benchmark
  16. ...and 25 more sections

Key Result

Lemma 1

Applying as:independent_metaas:meta_target_correlations to eq:multitask_kernel yields a sparse structure with the following non-zero entries of the coregionalization matrices where $\left[h_{v\theta}\right]_{oo'}$ are PSD in the basis $(o,o')$.

Figures (15)

  • Figure 1: High-level view of SMOG: Meta tasks are modeled independently of the target GP. The target GP models correlations between objectives and uses the weighted means and covariances of the meta GP to learn an informative target-task prior.
  • Figure 2: Example of a Sinusoidal function with two outputs (columns), three source tasks (rows 1--3), and one target task (row 4). SMOG learns a strong target-task posterior by leveraging meta tasks to learn an informative target-task prior, which is further refined by conditioning on the target data. See \ref{['subsec:benchmarks']} for details on the benchmark.
  • Figure 3: Performance of SMOG and competitors on the 6-dimensional two-objective Hartmann benchmark with 8 meta tasks and 64 observations per meta task.
  • Figure 4: Performance of SMOG and competitors on the Protein Structure Problem. SMOG and Ind.-ABLR show the best performance, with SMOG having a slight advantage at the end of the optimization loop.
  • Figure 5: Performance of SMOG and competitors on the two-objective Terrain benchmark. The results are averaged over different target tasks. SMOG shows competitive aggregated performance.
  • ...and 10 more figures

Theorems & Definitions (9)

  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Lemma 2
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • proof