Large Language Model-Based Evolutionary Optimizer: Reasoning with elitism

TL;DR

This work introduces Language-model Based Evolutionary Optimizer (LEO), a parameter-free, population-based framework that uses large language models to generate exploration and exploitation candidates for black-box optimization. By enforcing elitist guardrails through port-and-filter operations, LEO achieves competitive performance on 2D benchmarks, integrates with NSGA-II for multi-objective problems, and scales to higher dimensions and engineering tasks (nozzle design, heat transfer, wind-farm layout). The authors demonstrate LLMs’ reasoning capability in optimization and discuss practical remedies for hallucinations, variability, and high-dimensional challenges, while acknowledging substantial LLM-calling costs. Overall, LEO offers a modular, practical pathway to harnessing LLMs for diverse optimization tasks, with clear directions for improving robustness and scalability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable reasoning abilities, prompting interest in their application as black-box optimizers. This paper asserts that LLMs possess the capability for zero-shot optimization across diverse scenarios, including multi-objective and high-dimensional problems. We introduce a novel population-based method for numerical optimization using LLMs called Language-Model-Based Evolutionary Optimizer (LEO). Our hypothesis is supported through numerical examples, spanning benchmark and industrial engineering problems such as supersonic nozzle shape optimization, heat transfer, and windfarm layout optimization. We compare our method to several gradient-based and gradient-free optimization approaches. While LLMs yield comparable results to state-of-the-art methods, their imaginative nature and propensity to hallucinate demand careful handling. We provide practical guidelines for obtaining reliable answers from LLMs and discuss method limitations and potential research directions.

Figures (16)

• Figure 1: Convergence towards optimal nose cone shape with increasing number of decision variables as context.
• Figure 2: Performance of the LLM-based search algorithm (without a population) described in Section \ref{['llm1Dopt']} for the 2D Rosenbrock function.
• Figure 3: Schematic behind the LLM-assisted optimisation via exploration and exploitation of design space. Both figures illustrate the distribution of the initial pool of solution (triangle marker in blue color) randomly generated within the design space bounds along with the global optimal solution (in light green color). The figure(a) shows the overlay of the exploit pool of points (circle marker in sepia color) generated in close vicinity of the initial random solutions. The figure(b) shows the juxtaposition of the explore pool of solutions (diamond marker in sepia color) significantly away from the pool of initial randomly generated solutions. The best explore points (closest to the global optimal solution) is ported to exploit pool, following which the equal number of worst solutions from exploit pool will be removed. The $xz$ as well as $yz$ plane in both the figures depict the density distribution of the newly generated exploit and explore points.
• Figure 4: Schematic of the LEO framework.
• Figure 5: 2D benchmark functions and their convergence plots using LEO optimization.