Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heuristic Search as Language-Guided Program Optimization

Mingxin Yu, Ruixiao Yang, Chuchu Fan

TL;DR

This work proposes a structured framework for LLM-driven AHD that explicitly decomposes the heuristic discovery process into modular stages: a forward pass for evaluation, a backward pass for analytical feedback, and an update step for program refinement.

Abstract

Large Language Models (LLMs) have advanced Automated Heuristic Design (AHD) in combinatorial optimization (CO) in the past few years. However, existing discovery pipelines often require extensive manual trial-and-error or reliance on domain expertise to adapt to new or complex problems. This stems from tightly coupled internal mechanisms that limit systematic improvement of the LLM-driven design process. To address this challenge, we propose a structured framework for LLM-driven AHD that explicitly decomposes the heuristic discovery process into modular stages: a forward pass for evaluation, a backward pass for analytical feedback, and an update step for program refinement. This separation provides a clear abstraction for iterative refinement and enables principled improvements of individual components. We validate our framework across four diverse real-world CO domains, where it consistently outperforms baselines, achieving up to $0.17$ improvement in QYI on unseen test sets. Finally, we show that several popular AHD methods are restricted instantiations of our framework. By integrating them in our structured pipeline, we can upgrade the components modularly and significantly improve their performance.

Heuristic Search as Language-Guided Program Optimization

TL;DR

This work proposes a structured framework for LLM-driven AHD that explicitly decomposes the heuristic discovery process into modular stages: a forward pass for evaluation, a backward pass for analytical feedback, and an update step for program refinement.

Abstract

Large Language Models (LLMs) have advanced Automated Heuristic Design (AHD) in combinatorial optimization (CO) in the past few years. However, existing discovery pipelines often require extensive manual trial-and-error or reliance on domain expertise to adapt to new or complex problems. This stems from tightly coupled internal mechanisms that limit systematic improvement of the LLM-driven design process. To address this challenge, we propose a structured framework for LLM-driven AHD that explicitly decomposes the heuristic discovery process into modular stages: a forward pass for evaluation, a backward pass for analytical feedback, and an update step for program refinement. This separation provides a clear abstraction for iterative refinement and enables principled improvements of individual components. We validate our framework across four diverse real-world CO domains, where it consistently outperforms baselines, achieving up to improvement in QYI on unseen test sets. Finally, we show that several popular AHD methods are restricted instantiations of our framework. By integrating them in our structured pipeline, we can upgrade the components modularly and significantly improve their performance.
Paper Structure (55 sections, 9 equations, 3 figures, 4 tables)

This paper contains 55 sections, 9 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Automated Heuristic Design (Pre-LLM)
  4. LLM-Based Heuristic and Algorithm Discovery
  5. Method
  6. Problem Formulation: Heuristic Search as Program Optimization
  7. The LaGO Framework: Language-Guided Optimization
  8. Forward Pass (Evaluation).
  9. Backward Pass (Analyst).
  10. Update Step (Generator).
  11. Framework Instantiation
  12. Initialization.
  13. Step 1: Selection (Generator).
  14. Step 2: Directed Generation (Generator).
  15. Step 3: Candidate Evaluation (Forward Pass).
  16. ...and 40 more sections

Figures (3)

  • Figure 1: The LaGO framework for language-guided optimization. Our framework decomposes the automated heuristic design process into three modular stages: a forward pass (red), a backward pass (green), and an update step (blue). This modular structure supports systematic refinement of heuristic logic while preserving compatibility across different problem domains.
  • Figure 2: Component analysis of framework modules on PDPTW To isolate the performance gains contributed by specific components of our framework, we independently upgrade the baseline models with each proposed module, while keeping other modules the same as the original. Variant A introduces the co-evolution of constructive and refinement heuristics; variant B uses a code-writing analyst module; variant C applies a diversity-aware population management strategy; variant D adds the algorithm skeleton code to the generator prompt.
  • Figure 3: Convergence curves during training. Despite optimizing a larger joint search space, our diversity-aware sampling enables stable convergence to a better final fitness score than single-refinement baselines.