Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model

Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, Qingfu Zhang

TL;DR

EoH introduces an automatic heuristic design framework that evolves both the linguistic ideas of heuristics and their executable implementations using LLMs within an evolutionary loop. By employing five prompt strategies to guide thought-and-code co-evolution, EoH achieves state-of-the-art performance on online bin packing, TSP, and flow shop scheduling with far fewer LLM queries than prior methods. The study demonstrates the value of combining high-level reasoning with concrete code in a cooperative evolutionary process, including ablations showing the benefits of both representations and multiple prompts. The findings suggest practical potential for efficient automatic algorithm design and outline directions for domain-specific models and human-in-the-loop enhancements. Overall, EoH offers a principled, scalable path toward automatic generation of high-performance heuristics across diverse combinatorial problems.

Abstract

Heuristics are widely used for dealing with complex search and optimization problems. However, manual design of heuristics can be often very labour extensive and requires rich working experience and knowledge. This paper proposes Evolution of Heuristic (EoH), a novel evolutionary paradigm that leverages both Large Language Models (LLMs) and Evolutionary Computation (EC) methods for Automatic Heuristic Design (AHD). EoH represents the ideas of heuristics in natural language, termed thoughts. They are then translated into executable codes by LLMs. The evolution of both thoughts and codes in an evolutionary search framework makes it very effective and efficient for generating high-performance heuristics. Experiments on three widely studied combinatorial optimization benchmark problems demonstrate that EoH outperforms commonly used handcrafted heuristics and other recent AHD methods including FunSearch. Particularly, the heuristic produced by EoH with a low computational budget (in terms of the number of queries to LLMs) significantly outperforms widely-used human hand-crafted baseline algorithms for the online bin packing problem.

Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model

TL;DR

EoH introduces an automatic heuristic design framework that evolves both the linguistic ideas of heuristics and their executable implementations using LLMs within an evolutionary loop. By employing five prompt strategies to guide thought-and-code co-evolution, EoH achieves state-of-the-art performance on online bin packing, TSP, and flow shop scheduling with far fewer LLM queries than prior methods. The study demonstrates the value of combining high-level reasoning with concrete code in a cooperative evolutionary process, including ablations showing the benefits of both representations and multiple prompts. The findings suggest practical potential for efficient automatic algorithm design and outline directions for domain-specific models and human-in-the-loop enhancements. Overall, EoH offers a principled, scalable path toward automatic generation of high-performance heuristics across diverse combinatorial problems.

Abstract

Heuristics are widely used for dealing with complex search and optimization problems. However, manual design of heuristics can be often very labour extensive and requires rich working experience and knowledge. This paper proposes Evolution of Heuristic (EoH), a novel evolutionary paradigm that leverages both Large Language Models (LLMs) and Evolutionary Computation (EC) methods for Automatic Heuristic Design (AHD). EoH represents the ideas of heuristics in natural language, termed thoughts. They are then translated into executable codes by LLMs. The evolution of both thoughts and codes in an evolutionary search framework makes it very effective and efficient for generating high-performance heuristics. Experiments on three widely studied combinatorial optimization benchmark problems demonstrate that EoH outperforms commonly used handcrafted heuristics and other recent AHD methods including FunSearch. Particularly, the heuristic produced by EoH with a low computational budget (in terms of the number of queries to LLMs) significantly outperforms widely-used human hand-crafted baseline algorithms for the online bin packing problem.
Paper Structure (48 sections, 10 figures, 12 tables)

This paper contains 48 sections, 10 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Automatic Heuristic Design
  4. LLMs for Heuristic Design
  5. LLMs + EC
  6. Evolution of Heuristics (EoH)
  7. Main Idea
  8. Evolution Framework
  9. Heuristic Representation
  10. Prompt Strategies
  11. Initialization prompt
  12. Evolution prompts
  13. Experiments
  14. Experimental Settings
  15. Benchmarks and datasets.
  16. ...and 33 more sections

Figures (10)

  • Figure 1: Heuristic design often (a) relies on human expertise with reasoning over thoughts; recent progress has been made on (b) search over the space of codes; while (c) our method evolves both thoughts and codes using large language models.
  • Figure 2: Evolution of EoH for online bin packing. We outline the key thoughts and the corresponding code snippets of the best heuristics produced in some generations during evolution. We list the prompt strategies. We present the best heuristic in the final population and compare it with the best fit heuristic and the heuristic produced by FunSearch.
  • Figure 3: Convergence curves of EoH on TSP and FSSP. The y-axis represents the average gap (%) to the baseline and average makespan for TSP and FSSP, respectively. Each sample corresponds to a heuristic generated during the evolutionary process. The population size is 10 and the number of generations is 20. The gaps or makespans of some heuristics are out of the range of the figure or very close to each other. The red and orange lines depict the best and average performance of each population throughout the evolution.
  • Figure 4: Two examples of prompt engineering used in initialization prompt and E2 prompt for online bin packing.
  • Figure 5: Illustration of the generation of new heuristic and its code implementation using E2 prompt in one step in EoH on online bin packing problem. Five parent heuristics are selected from the population. E2 prompts LLM to first observe and summarize the common idea in the five heuristics, then design a new heuristic using this idea, and finally give a code implementation for the designed heuristic.
  • ...and 5 more figures