Automated Algorithm Design for Auto-Tuning Optimizers

TL;DR

Auto-tuning suffers from vast, irregular search spaces that defy manual exploration. The paper introduces LLaMEA, a framework that uses large language models to automatically design optimization algorithms tailored to auto-tuning tasks, with evaluation integrated into the Kernel Tuner workflow and guided by a robust performance metric $ \mathcal{P}$ under a budget defined by the baseline’s 95th percentile progress. On four real-world auto-tuning kernels across six GPUs, LLM-generated optimizers achieve meaningful improvements from extra information prompts, rivaling and sometimes surpassing human-designed optimizers, with the best cases showing about $72.4\%$ higher performance than baselines. This work demonstrates a paradigm shift toward automated optimizer design for auto-tuning, offering open-source implementations and demonstrating substantial practical impact for HPC kernel tuning. The approach highlights both the feasibility and benefits of problem-specific optimizer synthesis using LLMs in complex, heterogeneous hardware settings.

Abstract

Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular parameter spaces make manual exploration infeasible. Traditionally, auto-tuning relies on well-established optimization algorithms such as evolutionary algorithms, annealing methods, or surrogate model-based optimizers to efficiently find near-optimal configurations. However, designing effective optimizers remains challenging, as no single method performs best across all tuning tasks. In this work, we explore a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search-space characteristics results to produce specialized optimization strategies, which are iteratively examined and improved. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in optimization algorithms of two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7\% and 14.6\%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving, on average, 72.4\% improvement over state-of-the-art optimizers for auto-tuning.

Figures (8)

• Figure 1: Overview of our designed system, showing how LLaMEA generates optimization algorithms, which are evaluated using the autotuning methodology, which tests the performance of the optimization algorithm in Kernel Tuner.
• Figure 2: Overview of the software architecture of Kernel Tuner.
• Figure 3: Prompt template used to generate initial candidate optimization algorithms.
• Figure 4: Mutation prompts used by LLaMEA for Kernel Tuner.
• Figure 5: The aggregate performance over time over all search spaces of the application-specific optimization algorithms, each with and without additional search space knowledge. Mean of 100 runs each, the shaded area marks the 95% confidence interval.