A Systematic Survey on Large Language Models for Algorithm Design

TL;DR

This work synthesizes the emergent field of large language models for algorithm design (LLM4AD) by delivering the first systematic review, a comprehensive four-dimensional taxonomy (LLM Roles, Search Methods, Prompt Methods, Applications), and a cross-cutting analysis of 180+ papers. It frameworks how LLMs augment optimization, ML, science discovery, and industry through four roles (optimizers, predictors, extractors, designers) and diverse search strategies, while detailing prompting techniques and application domains. The paper also identifies key challenges—scalability, interpretability, security, cost, and idea innovation—and outlines promising directions, including domain- and multi-modal LLMs and benchmarking efforts. Overall, it provides a structured blueprint for researchers and practitioners to navigate, evaluate, and advance LLMaD research and practice.

Abstract

Algorithm Design (AD) is crucial for effective problem-solving across various domains. The advent of Large Language Models (LLMs) has notably enhanced the automation and innovation within this field, offering new perspectives and promising solutions. Over the past three years, the integration of LLMs into AD (LLM4AD) has seen substantial progress, with applications spanning optimization, machine learning, mathematical reasoning, and scientific discovery. Given the rapid advancements and expanding scope of this field, a systematic review is both timely and necessary. This paper provides a systematic review of LLM4AD. First, we offer an overview and summary of existing studies. Then, we introduce a taxonomy and review the literature across four dimensions: the roles of LLMs, search methods, prompt methods, and application domains with a discussion of potential and achievements of LLMs in AD. Finally, we identify current challenges and highlight several promising directions for future research.

Figures (9)

• Figure 1: Four stages for paper collection.
• Figure 2: Overview on LLM4AD papers.
• Figure 3: The word cloud is generated from the titles and abstracts of all reviewed papers, with each word appearing at least five times. It features the top 80 keywords, organized into four color-coded clusters.
• Figure 4: A four-dimensional taxonomy.
• Figure 5: Large Language Models as Optimizers (LLMaO). LLMs serve as optimizers within the algorithm to generate new solutions. This typically involves using the LLM in an iterative search process to enhance solution quality. Here, the algorithm and its parameters are usually crafted by humans.