Robust Heuristic Algorithm Design with LLMs

TL;DR

The paper addresses the vulnerability of heuristic designs to edge-cases by proposing Robusta, a framework that augments LLM-based design with explanations and region-based specialization. It introduces a two-stage pipeline that analyzes where a base heuristic underperforms, generates region-specific explanations, and uses those explanations as suggestions to evolve specialized heuristics per region, forming an ensemble. Empirical results on traffic engineering demonstrate dramatic improvements in worst-case (approximately 28×) and substantial gains in average-case performance with runtime comparable to the baseline, validating the robustness-focused approach. This work highlights the potential of combining explanation-driven search with regional specialization to create more reliable heuristics and outlines concrete research directions for scaling, intent alignment, and broader networking deployment.

Abstract

We posit that we can generate more robust and performant heuristics if we augment approaches using LLMs for heuristic design with tools that explain why heuristics underperform and suggestions about how to fix them. We find even simple ideas that (1) expose the LLM to instances where the heuristic underperforms; (2) explain why they occur; and (3) specialize design to regions in the input space, can produce more robust algorithms compared to existing techniques~ -- ~the heuristics we produce have a $\sim28\times$ better worst-case performance compared to FunSearch, improve average performance, and maintain the runtime.

Figures (12)

• Figure 1: Robusta uses heuristic analyzers in a loop with the LLM. The heuristic analyzer finds: (a) regions of the input space where the heuristic underperforms ($\mathcal{R}_1$, $\mathcal{R}_2$); (b) explanations to guide the search. For each region, our solution uses the LLM to suggest how to improve the heuristic for that region and then implements it based on those suggestions. We can run ($\mathcal{H}_1$, $\mathcal{H}_2$) for inputs that come from their corresponding region.
• Figure 2: We compare FunSearch with solutions where we (1) focus on adversarial samples in each step (\ref{['subsec:adversarial']}); and (2) also incorporate "suggestions" in each step (\ref{['sec:explanations']}); on a traffic engineering problem. Dashed lines report the lowest (best) suboptimality achieved for a heuristic on training samples so far, and the solid lines measure it for a held-out set of adversarial samples. The suboptimality is the worst-case performance of a heuristic compared to the optimal across all samples.
• Figure 3: we create an approach which uses FunSearch to create specialized heuristics for each region (separate regions) with one that generates a single heuristic to operate over the entire space. We find specialized heuristics outperform monolithic ones. The dashed and solid lines show overall suboptimality so far on training and held-out samples, respectively.
• Figure 4: Explanations benefit the search. We consider one step of the search process and see that we can find better heuristics if we first "explain" why a heuristic underperforms. We can improve the approach further if we convert these explanations into suggestions on how to improve the heuristic. We evaluate all of the approaches on a held-out set of adversarial samples.
• Figure 5: We evaluate our ideas end-to-end. We create specialized heuristics on $5$ regions and use approximate suggestions. Robusta finds heuristics with better worst-case performance.