Tuning the Tuner: Introducing Hyperparameter Optimization for Auto-Tuning

TL;DR

The paper tackles the underexplored impact of hyperparameters on auto-tuning performance and introduces a general, statistically robust framework to tune these hyperparameters across diverse auto-tuning problems. It combines a formalized objective and robust performance metric with a simulation mode and FAIR benchmark data to enable scalable, reproducible evaluation, integrated within the Kernel Tuner. Empirical results show substantial gains, with average improvements up to $94.8\%$ across training spaces and even larger gains ($204.7\%$) when using meta-strategies and extended hyperparameter tuning, while the simulation mode delivers about $130\times$ speedups in tuning cost. Overall, the work demonstrates that tuning the tuners themselves can significantly advance auto-tuning practice and provides valuable resources to support reproducible, scalable research in this domain.

Abstract

Automatic performance tuning (auto-tuning) is widely used to optimize performance-critical applications across many scientific domains by finding the best program variant among many choices. Efficient optimization algorithms are crucial for navigating the vast and complex search spaces in auto-tuning. As is well known in the context of machine learning and similar fields, hyperparameters critically shape optimization algorithm efficiency. Yet for auto-tuning frameworks, these hyperparameters are almost never tuned, and their potential performance impact has not been studied. We present a novel method for general hyperparameter tuning of optimization algorithms for auto-tuning, thus "tuning the tuner". In particular, we propose a robust statistical method for evaluating hyperparameter performance across search spaces, publish a FAIR data set and software for reproducibility, and present a simulation mode that replays previously recorded tuning data, lowering the costs of hyperparameter tuning by two orders of magnitude. We show that even limited hyperparameter tuning can improve auto-tuner performance by 94.8% on average, and establish that the hyperparameters themselves can be optimized efficiently with meta-strategies (with an average improvement of 204.7%), demonstrating the often overlooked hyperparameter tuning as a powerful technique for advancing auto-tuning research and practice.

Figures (9)

• Figure 1: The hyperparameter tuning pipeline for auto-tuning. Kernel Tuner's hyperparameter tuning functionality (the outermost layer) calls the autotuning methodology software to get an aggregate performance score, which is obtained by running the optimization algorithm on various search spaces.
• Figure 2: Violin plots of the performance scores (higher is better) for all hyperparameter configurations of the evaluated optimization algorithms, showing the mean (white line), boxplot (black box), and distribution (area).
• Figure 3: Best and worst scores on tuning, training (re-executed for comparison), and test for evaluated optimization algorithms.
• Figure 4: Impact of tuning on optimization algorithm performance; left-hand and right-hand columns show the performance on all search spaces for suboptimal and optimal optimization algorithm versions, respectively.
• Figure 5: Aggregate performance over time between optimization algorithms with mean and optimal hyperparameters across all search spaces.