In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics

TL;DR

This work tackles the high cost of LLM-driven algorithm design by introducing LLaMEA-HPO, a hybrid framework that delegates numerical hyper-parameter tuning to SMAC3 while the LLM concentrates on generating novel algorithmic structures. By separating structural search from parameter optimization, the approach reduces the number of costly LLM queries and improves efficiency without compromising solution quality on benchmark tasks. Empirical results across Online Bin Packing, BBOB AOCC, and TSP with GLS show favorable performance with substantially fewer prompts, demonstrating the practical impact of coupling LLM-based design with dedicated HPO. Overall, the work highlights a scalable pathway for efficient LLM-based code generation in computationally intensive optimization settings.

Abstract

Large Language Models (LLMs) have shown great potential in automatically generating and optimizing (meta)heuristics, making them valuable tools in heuristic optimization tasks. However, LLMs are generally inefficient when it comes to fine-tuning hyper-parameters of the generated algorithms, often requiring excessive queries that lead to high computational and financial costs. This paper presents a novel hybrid approach, LLaMEA-HPO, which integrates the open source LLaMEA (Large Language Model Evolutionary Algorithm) framework with a Hyper-Parameter Optimization (HPO) procedure in the loop. By offloading hyper-parameter tuning to an HPO procedure, the LLaMEA-HPO framework allows the LLM to focus on generating novel algorithmic structures, reducing the number of required LLM queries and improving the overall efficiency of the optimization process. We empirically validate the proposed hybrid framework on benchmark problems, including Online Bin Packing, Black-Box Optimization, and the Traveling Salesperson Problem. Our results demonstrate that LLaMEA-HPO achieves superior or comparable performance compared to existing LLM-driven frameworks while significantly reducing computational costs. This work highlights the importance of separating algorithmic innovation and structural code search from parameter tuning in LLM-driven code optimization and offers a scalable approach to improve the efficiency and effectiveness of LLM-based code generation.

Figures (14)

• Figure 1: The summary of the proposed LLM driven algorithm design with Hyper-parameter optimization framework LLaMEA-HPO. Marked in violet are the components related to HPO.
• Figure 2: Convergence curves (lower is better) of the LLaMEA-HPO algorithm and the EoH algorithm on Online Bin Packing Problems (BP). Individual runs are denoted by dotted lines, the average convergence by a solid line and the standard deviation by the shaded area. The left plot shows the convergence with the number of LLM prompts used on the X-axis, and the right plot shows the convergence with the number of full benchmark evaluations used (including HPO evaluations).
• Figure 3: Convergence curves of the mean AOCC (higher is better) of the LLaMEA-HPO algorithm and the EoH algorithm on $5D$ BBOB problems. The average convergence over $5$ individual runs are denoted by a solid line and the standard deviation by the shaded area. The left plot shows the convergence with the number of LLM prompts used on the X-axis, and the right plot shows the convergence with the number of full benchmark evaluations used (including HPO instance evaluations).
• Figure 4: Convergence curves for LLaMEA-HPO (dotted lines are individual runs) and the non-hyper-parameter optimized algorithms as baseline (LLaMEA-HPO default).
• Figure 5: Glicko-2 glickman2012example rating (higher is better) of each "QEAD_MS", "QIDE_AN" and "QuantumEnhancedADEES", the three hyper-parameter tuned algorithms resulting from three independent runs of LLaMEA-HPO, versus the original LLaMEA generated algorithm "ERADS_QuantumFluxUltraRefined" and the best EoH generated algorithm (denoted "EoH"). Glicko2 uses 200 matches, picking a random run at the specified budget of $10\,000$ per BBOB function between pairs of algorithms.