LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AI

TL;DR

This work addresses the bottleneck in automated algorithm design where LLM-driven discovery relies predominantly on fitness feedback. It introduces LLaMEA-SAGE, a structured-feedback mechanism that derives AST-based code features and trains an archive-based surrogate to identify performance-affecting features via SHAP attributions, translating them into natural-language mutation guidance for LLMs. The approach preserves template-free, end-to-end algorithm discovery while biasing exploration toward architecture- and complexity-related code properties that correlate with performance. Empirical results on SBOX-COST and the MA-BBOB MA benchmark show faster convergence and competitive performance against state-of-the-art baselines, supported by code-evolution analyses and token-cost considerations. Overall, the work demonstrates that interpretable, code-level signals can effectively guide LLM-driven algorithm evolution, offering a principled pathway to more efficient and scalable automated algorithm design.

Abstract

Large language models have enabled automated algorithm design (AAD) by generating optimization algorithms directly from natural-language prompts. While evolutionary frameworks such as LLaMEA demonstrate strong exploratory capabilities across the algorithm design space, their search dynamics are entirely driven by fitness feedback, leaving substantial information about the generated code unused. We propose a mechanism for guiding AAD using feedback constructed from graph-theoretic and complexity features extracted from the abstract syntax trees of the generated algorithms, based on a surrogate model learned over an archive of evaluated solutions. Using explainable AI techniques, we identify features that substantially affect performance and translate them into natural-language mutation instructions that steer subsequent LLM-based code generation without restricting expressivity. We propose LLaMEA-SAGE, which integrates this feature-driven guidance into LLaMEA, and evaluate it across several benchmarks. We show that the proposed structured guidance achieves the same performance faster than vanilla LLaMEA in a small controlled experiment. In a larger-scale experiment using the MA-BBOB suite from the GECCO-MA-BBOB competition, our guided approach achieves superior performance compared to state-of-the-art AAD methods. These results demonstrate that signals derived from code can effectively bias LLM-driven algorithm evolution, bridging the gap between code structure and human-understandable performance feedback in automated algorithm design.

Figures (16)

• Figure 1: Overview of the proposed method LLaMEA-SAGE.
• Figure 2: Average best-so-far fitness (AOCC) over time of the baseline LLaMEA and the proposed code-feature guided approach (LLaMEA-SAGE) on the SBOX-COST suite. Aggregated over $5$ independent runs, the 95% confidence interval is shown as shaded area.
• Figure 3: Average speed-up of LLaMEA-SAGE versus vanilla LLaMEA. The speed-up is how much faster than LLaMEA the proposed method is, i.e. if LLaMEA-SAGE reaches fitness value $f_i$ in $m$ evaluations and LLaMEA reaches that same fitness value in $n$ evaluations, the speed-up is $n/m$.
• Figure 4: Code features and direction used during the $5$ different runs to guide the automated design of black-box optimization algorithms.
• Figure 5: Consistency between code feature--guided suggestions and actual code feature mutation by LLaMEA-SAGE for different mutation prompts (left the random new and right the refine prompt).