When Large Language Model Meets Optimization

TL;DR

This survey analyzes the bidirectional relationship between optimization algorithms and large language models (LLMs), highlighting how LLMs can function as both black-box search operators and generators of optimization methods. It also reviews how optimization techniques can tailor LLMs through prompt engineering, architectural search, and multi-task optimization to improve efficiency, robustness, and applicability. Key contributions include a taxonomy of LLM–OA hybrids, representative methods (e.g., OPRO, OptiMUS, AEL, ReEvo, LEO, NAS-assisted approaches), and examples across software programming, neural architecture search, and content generation. The article concludes with forward-looking directions on theoretical foundations, automated intelligent optimization, robustness, and cross-domain impact.

Abstract

Optimization algorithms and large language models (LLMs) enhance decision-making in dynamic environments by integrating artificial intelligence with traditional techniques. LLMs, with extensive domain knowledge, facilitate intelligent modeling and strategic decision-making in optimization, while optimization algorithms refine LLM architectures and output quality. This synergy offers novel approaches for advancing general AI, addressing both the computational challenges of complex problems and the application of LLMs in practical scenarios. This review outlines the progress and potential of combining LLMs with optimization algorithms, providing insights for future research directions.

Figures (3)

• Figure 1: The split of LLMs assist optimization algorithms.
• Figure 2: OAs methodological framework for optimizing LLMs
• Figure 3: The split of LLMs assist optimization algorithms.