Practitioner Motives to Use Different Hyperparameter Optimization Methods

TL;DR

<3-5 sentence high-level summary> This study investigates why practitioners choose different hyperparameter optimization methods, despite the superior sample efficiency of automated approaches like Bayesian optimization. Using 20 interviews and a 49-participant online survey, it identifies six primary goals and fourteen contextual factors that shape method selection, revealing that decisions are not driven solely by performance but also by compute constraints, effort, and the desire to understand models. The authors provide a conceptual foundation and a mapping of goals, methods, and contexts to inform user-centered AutoML tool design, and they discuss perceived success across method–goal pairs to guide future tool development. The work highlights the need for more transparent, explainable HPO tools that integrate practitioner knowledge and support human–in–the–loop decision making in real-world workflows.

Abstract

Programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization and evolutionary algorithms, are highly sample-efficient in identifying optimal hyperparameter configurations for machine learning (ML) models. However, practitioners frequently use less efficient methods, such as grid search, which can lead to under-optimized models. We suspect this behavior is driven by a range of practitioner-specific motives. Practitioner motives, however, still need to be clarified to enhance user-centered development of HPO tools. To uncover practitioner motives to use different HPO methods, we conducted 20 semi-structured interviews and an online survey with 49 ML experts. By presenting main goals (e.g., increase ML model understanding) and contextual factors affecting practitioners' selection of HPO methods (e.g., available computer resources), this study offers a conceptual foundation to better understand why practitioners use different HPO methods, supporting development of more user-centered and context-adaptive HPO tools in automated ML.

Figures (9)

• Figure 1: Overview of HPO methods used by 49 study participants.
• Figure 2: Relative frequency of pursued goals by 49 study participants.
• Figure 3: Frequency of goal and HPO method combinations. Per cell, all presented values are normalized to the number of participants having applied the corresponding HPO method.
• Figure 4: Percentage of participants that incorporated the individual contextual factors.
• Figure 5: Overview of the average self-perceived relevance of contextual factors. Results are reported on a scale from 0 (very low) to 5 (very high). On average, no contextual factor was rated with a higher relevance than 3, which is why we masked out the values 4 and 5. Blue lines indicate error bars of one standard deviation.