Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview

TL;DR

This paper surveys multi-objective hyperparameter optimization (MOHPO) for supervised ML, arguing that many practical ML systems must balance predictive performance with efficiency, fairness, interpretability, robustness, and sparsity. It formalizes MOHPO as a black-box, potentially noisy optimization over mixed hierarchies, and reviews foundational concepts (Pareto optimality, evaluation, normalization) alongside a broad set of optimization methods, including scalarization, evolutionary algorithms, Bayesian optimization, and multi-fidelity approaches. The work details prominent MOEAs (NSGA-II, MOEA/D, SMS-EMOA), multi-objective BO variants (ParEGO, EHI, SMS-EGO, MESMO/PESMO), and recent software tooling, with discussions on benchmarks and the practical performance of methods across ML tasks. It further maps MOHPO to concrete ML objectives (prediction accuracy, efficiency, fairness, interpretability, robustness, and sparsity) and industry relevance, highlighting open challenges such as preference integration, noisy evaluations, realistic evaluation, and the need for standardized MOHPO benchmarks. Overall, the paper provides a comprehensive framework and practical guidance for applying MOHPO to real-world ML pipelines and AutoML contexts, emphasizing that a Pareto-front view enables more transparent, task-specific trade-offs than single-metric optimization.

Abstract

Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization. We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness.

Figures (9)

• Figure 1: Taxonomy of common black-box optimization problems. Attributes that are related to MOHPO - and therefore a substantial focus in this paper - are highlighted in green.
• Figure 2: Illustration for a two-dimensional MOO problem with two objectives $c_1$ and $c_2$. The left plot shows the search space $\tilde{\Lambda}$, and the right plot shows the objective space $\vmathbb{R}^2$. Non-dominated configurations - the estimated Pareto set (left) - and their mapping to the co-domain - the estimated Pareto front (right) - are highlighted.
• Figure 3: The plot shows the hypervolume indicator (the area of the blue shaded region) regarding the reference point $(1,1)$ (marked in red).
• Figure 4: Basic loop of an evolutionary algorithm.
• Figure 5: Exemplary bracket run (figure inspired by automl_book). Faint lines represent future performance of HPCs that were discarded early.