Hyperparameter Importance Analysis for Multi-Objective AutoML

TL;DR

The paper tackles the challenge of assessing hyperparameter importance in multi-objective AutoML by proposing a weighted $HPI$ framework that scalarizes objectives along Pareto-front tradeoffs. It develops two surrogate-based methods, $MO$-$fANOVA$ and $MO$-ablation path analysis, to quantify hyperparameter influence across different weightings, supported by data preparation and a Pareto-based weighting scheme. Through three MO-HPO experiments on MNIST, Adult, and CIFAR-10, the authors demonstrate that hyperparameters can exhibit distinct, tradeoff-dependent importance, and that the two methods offer complementary, actionable insights. The work advances human-centered MO AutoML by enabling more nuanced hyperparameter tuning and interpretability, while acknowledging limitations and outlining directions for broader objective setups and tool integration.

Abstract

Hyperparameter optimization plays a pivotal role in enhancing the predictive performance and generalization capabilities of ML models. However, in many applications, we do not only care about predictive performance but also about additional objectives such as inference time, memory, or energy consumption. In such multi-objective scenarios, determining the importance of hyperparameters poses a significant challenge due to the complex interplay between the conflicting objectives. In this paper, we propose the first method for assessing the importance of hyperparameters in multi-objective hyperparameter optimization. Our approach leverages surrogate-based hyperparameter importance measures, i.e., fANOVA and ablation paths, to provide insights into the impact of hyperparameters on the optimization objectives. Specifically, we compute the a-priori scalarization of the objectives and determine the importance of the hyperparameters for different objective tradeoffs. Through extensive empirical evaluations on diverse benchmark datasets with three different objective pairs, each combined with accuracy, namely time, demographic parity loss, and energy consumption, we demonstrate the effectiveness and robustness of our proposed method. Our findings not only offer valuable guidance for hyperparameter tuning in multi-objective optimization tasks but also contribute to advancing the understanding of hyperparameter importance in complex optimization scenarios.

Figures (4)

• Figure 1: Overview of the MO-fANOVA method. On the left, the importance of each hyperparameter for Objective 1 is shown. On the right, our extension for the importance of each hyperparameter for different weightings of the objectives is displayed exemplarily.
• Figure 2: Overview of the MO-ablation path analysis. On the left, an exemplary Pareto front is displayed, with several ablation paths going from the default configuration to different configurations on the Pareto front. Every path is associated with a weighting of the objectives and thus gives a different value for the difference in performance per hyperparameter. We convert this to the plot on the right, where the total performance for different weightings is displayed as a stacked plot of hyperparameter contributions.
• Figure 3: Results for the time experiment. The Pareto front is on the left (error vs. training time in seconds), with the red dot being the default performance. The MO-fANOVA results are in the middle, and the MO-ablation path analysis is on the right. The x-axis corresponds to the weighting of the minimum error objective.
• Figure 4: Results for the fairness experiment. The Pareto front is on the left (error vs. demographic parity loss), with the red dot being the default performance. The MO-fANOVA results are in the middle, and the MO-ablation path analysis is on the right. The x-axis corresponds to the weighting of the minimum error objective.