Advancing Automated Algorithm Design via Evolutionary Stagewise Design with LLMs

TL;DR

EvoStage is introduced, a novel evolutionary paradigm that bridges the gap between the rigorous demands of industrial-scale algorithm design and the LLM-based algorithm design methods, and outperforms human-expert designs and existing LLM-based methods within only a couple of evolution steps.

Abstract

With the rapid advancement of human science and technology, problems in industrial scenarios are becoming increasingly challenging, bringing significant challenges to traditional algorithm design. Automated algorithm design with LLMs emerges as a promising solution, but the currently adopted black-box modeling deprives LLMs of any awareness of the intrinsic mechanism of the target problem, leading to hallucinated designs. In this paper, we introduce Evolutionary Stagewise Algorithm Design (EvoStage), a novel evolutionary paradigm that bridges the gap between the rigorous demands of industrial-scale algorithm design and the LLM-based algorithm design methods. Drawing inspiration from CoT, EvoStage decomposes the algorithm design process into sequential, manageable stages and integrates real-time intermediate feedback to iteratively refine algorithm design directions. To further reduce the algorithm design space and avoid falling into local optima, we introduce a multi-agent system and a "global-local perspective" mechanism. We apply EvoStage to the design of two types of common optimizers: designing parameter configuration schedules of the Adam optimizer for chip placement, and designing acquisition functions of Bayesian optimization for black-box optimization. Experimental results across open-source benchmarks demonstrate that EvoStage outperforms human-expert designs and existing LLM-based methods within only a couple of evolution steps, even achieving the historically state-of-the-art half-perimeter wire-length results on every tested chip case. Furthermore, when deployed on a commercial-grade 3D chip placement tool, EvoStage significantly surpasses the original performance metrics, achieving record-breaking efficiency. We hope EvoStage can significantly advance automated algorithm design in the real world, helping elevate human productivity.

Figures (19)

• Figure 1: The overview of EvoStage. The upper left part is the intuitive illustration of the local perspective operator (i.e., the Stagewise-Design operator), where the algorithm design task is decomposed automatically by the LLM coordinator agent, with the coordinator reflecting on the intermediate information (denoted as "Inter layout", using chip placement as an example) and giving guidance for the next stage, and $N$ LLM coder agents producing $N$ algorithm components (one coder responsible for one respective component) according to the guidance from the coordinator stage-by-stage. The lower left part is the intuitive illustration of the two global perspective operators. The Global-Explore operator is to prompt the coder to produce a novel multi-stage heuristic in one shot with a selected reference, while the Global-Enhance operator is to prompt the coder to produce a better multi-stage heuristic in one shot with a selected reference. The right part is the multi-stage heuristic population, which maintains $M$ algorithms. Each algorithm is divided into $N$ components, each of which is implemented as a multi-stage heuristic. When selecting an algorithm from the population, we select the corresponding $N$ multi-stage heuristics for the $N$ components and assemble a complete algorithm from them. EvoStage performs an iterative process of selecting parent algorithms from the population, reproducing new offspring algorithms by operators, and updating the population to keep better algorithms, with the goal of iteratively improving the algorithms maintained in the population and finally generating a high-performing algorithm.
• Figure 2: The intuitive comparison between the traditional black-box modeling of algorithm design (upper) and the proposed Stagewise Design paradigm (lower).
• Figure 3: One detailed example of the reflection reasoning of the coordinator between stages in the Stagewise Design paradigm.
• Figure 4: The example prompt of the local perspective operator Stagewise-Design.
• Figure 5: The example prompt of the two global perspective operators: Global-Explore and Global-Enhance.