Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Design

Chentong Chen, Mengyuan Zhong, Ye Fan, Jialong Shi, Jianyong Sun

TL;DR

The paper addresses the challenge of designing effective heuristics for combinatorial optimization ($CO$) problems, which are $NP$-hard, by integrating Large Language Models (LLMs) with Evolutionary Computation (EC) through HiFo-Prompt. This framework introduces a hierarchical prompting architecture with a Foresight module (Evolutionary Navigator) for real-time, state-aware guidance and a Hindsight module (Insight Pool) for accumulating reusable design principles, enabling the LLM to learn from experience without opaque fine-tuning. A self-evolving loop distills successful principles into actionable prompts and dynamically adapts the search strategy, reducing training time and query costs. Empirical results on TSP, Online BPP, Flow Shop Scheduling, and Bayesian Optimization demonstrate state-of-the-art performance and strong sample efficiency, often achieving superior solutions with hundreds of LLM calls, highlighting the framework’s potential for cross-domain algorithmic discovery.

Abstract

LLM-based Automatic Heuristic Design (AHD) within Evolutionary Computation (EC) frameworks has shown promising results. However, its effectiveness is hindered by the use of static operators and the lack of knowledge accumulation mechanisms. We introduce HiFo-Prompt, a framework that guides LLMs with two synergistic prompting strategies: Foresight and Hindsight. Foresight-based prompts adaptively steer the search based on population dynamics, managing the exploration-exploitation trade-off. In addition, hindsight-based prompts mimic human expertise by distilling successful heuristics from past generations into fundamental, reusable design principles. This dual mechanism transforms transient discoveries into a persistent knowledge base, enabling the LLM to learn from its own experience. Empirical results demonstrate that HiFo-Prompt significantly outperforms state-of-the-art LLM-based AHD methods, generating higher-quality heuristics while achieving substantially faster convergence and superior query efficiency. Our code is available at https://github.com/Challenger-XJTU/HiFo-Prompt.

HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Design

TL;DR

The paper addresses the challenge of designing effective heuristics for combinatorial optimization () problems, which are -hard, by integrating Large Language Models (LLMs) with Evolutionary Computation (EC) through HiFo-Prompt. This framework introduces a hierarchical prompting architecture with a Foresight module (Evolutionary Navigator) for real-time, state-aware guidance and a Hindsight module (Insight Pool) for accumulating reusable design principles, enabling the LLM to learn from experience without opaque fine-tuning. A self-evolving loop distills successful principles into actionable prompts and dynamically adapts the search strategy, reducing training time and query costs. Empirical results on TSP, Online BPP, Flow Shop Scheduling, and Bayesian Optimization demonstrate state-of-the-art performance and strong sample efficiency, often achieving superior solutions with hundreds of LLM calls, highlighting the framework’s potential for cross-domain algorithmic discovery.

Abstract

LLM-based Automatic Heuristic Design (AHD) within Evolutionary Computation (EC) frameworks has shown promising results. However, its effectiveness is hindered by the use of static operators and the lack of knowledge accumulation mechanisms. We introduce HiFo-Prompt, a framework that guides LLMs with two synergistic prompting strategies: Foresight and Hindsight. Foresight-based prompts adaptively steer the search based on population dynamics, managing the exploration-exploitation trade-off. In addition, hindsight-based prompts mimic human expertise by distilling successful heuristics from past generations into fundamental, reusable design principles. This dual mechanism transforms transient discoveries into a persistent knowledge base, enabling the LLM to learn from its own experience. Empirical results demonstrate that HiFo-Prompt significantly outperforms state-of-the-art LLM-based AHD methods, generating higher-quality heuristics while achieving substantially faster convergence and superior query efficiency. Our code is available at https://github.com/Challenger-XJTU/HiFo-Prompt.

Paper Structure

This paper contains 79 sections, 24 equations, 4 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-driven Automatic Heuristic Design
  4. Knowledge Management in Generative Search
  5. Adaptive Control in Evolutionary Computation
  6. Methodology
  7. Guided Prompt Synthesis for AHD
  8. Foundational prompt strategies.
  9. Hindsight Module.
  10. Foresight Module.
  11. Hindsight: Mechanisms of the Self-Evolving Insight Pool
  12. Insight Extraction and Admission.
  13. Insight Retrieval and Credit Assignment.
  14. Foresight: The Evolutionary Navigator for State-Aware Guidance
  15. Experiments
  16. ...and 64 more sections

Figures (4)

  • Figure 1: The framework of HiFo-Prompt, which comprises two core processes. (1) Prompt Construction: A foundational prompt is dynamically augmented with design directives from the Foresight and insights from the Hindsight to form the final composite prompt. (2) Knowledge Evolution: A self-evolving loop is established where elite heuristics from the evolutionary process are distilled into new insights, continuously enriching the Hindsight module's knowledge base.
  • Figure 2: Dynamic prompt generation process of HiFo-Prompt.
  • Figure 3: A Key Example of Convergence Analysis for TSP and Online BPP.
  • Figure 4: Parameter sensitivity analysis of population size, maximum generations, insight pool size, stagnation threshold and diversity threshold.