Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization

TL;DR

This work tackles the challenge of multi-objective hyperparameter optimization (HPO) where the relevance of hyperparameters shifts with objective trade-offs. It introduces HPI-ParEGO, a dynamic approach that uses HyperSHAP to estimate MO-HPI under the current scalarization and reduces the configuration space by fixing unimportant hyperparameters, thereby concentrating search on influential dimensions. The method adapts the thresholding over the course of optimization and demonstrates faster convergence and improved Pareto-front quality on synthetic PyMOO tasks and real-world YAHPO-Gym benchmarks, with thorough ablations validating key design choices. Overall, the approach offers a data-driven path to more resource-efficient MO-HPO, with potential benefits for Green AutoML, while acknowledging limitations in surrogate accuracy and scalability for very large hyperparameter spaces.

Abstract

Choosing a suitable ML model is a complex task that can depend on several objectives, e.g., accuracy, model size, fairness, inference time, or energy consumption. In practice, this requires trading off multiple, often competing, objectives through multi-objective optimization (MOO). However, existing MOO methods typically treat all hyperparameters as equally important, overlooking that hyperparameter importance (HPI) can vary significantly depending on the trade-off between objectives. We propose a novel dynamic optimization approach that prioritizes the most influential hyperparameters based on varying objective trade-offs during the search process, which accelerates empirical convergence and leads to better solutions. Building on prior work on HPI for MOO post-analysis, we now integrate HPI, calculated with HyperSHAP, into the optimization. For this, we leverage the objective weightings naturally produced by the MOO algorithm ParEGO and adapt the configuration space by fixing the unimportant hyperparameters, allowing the search to focus on the important ones. Eventually, we validate our method with diverse tasks from PyMOO and YAHPO-Gym. Empirical results demonstrate improvements in convergence speed and Pareto front quality compared to baselines.

Figures (13)

• Figure 1: The plot shows the HPO task lcbench_12605 for one seed, trading off accuracy vs time, the top plot compares ParEGO with our improved variant with dynamic HPI consideration, the middle shows changing HPI for the different scalarizations, which are shown in the bottom. There is an initialization and a convergence phase, during which no configuration space reduction is performed.
• Figure 2: Algorithm overview: Blue corresponds to the normal Bayesian optimization (BO) loop, orange to the ParEGO algorithm, and yellow to the HPI-ParEGO additions proposed in this work.
• Figure 3: Results of the HPI-ParEGO optimizer compared to all baselines on the PyMOO tasks.
• Figure 4: Comparing the HPI-ParEGO optimizer to the ParEGO baseline on PyMOO per task. Values $> 0$ indicate a better performance of HPI-ParEGO.
• Figure 5: Ablation results overview of all settings.