Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Existing Optimization Algorithms with LLMs

Camilo Chacón Sartori, Christian Blum

TL;DR

The paper investigates whether LLMs can improve existing optimization algorithms by treating LLMs as assistants that suggest novel heuristics and implementation improvements. It uses an expert CMSA implementation for the Maximum Independent Set (MIS) problem as a testbed and demonstrates that an LLM-generated heuristic (LLM-CMSA-V1) outperforms the expert-designed CMSA, with larger gains on bigger, denser graphs. It extends the approach with an entropy-based diversification (LLM-CMSA-V2) and subsequent code-optimization variants (V1-PERF, V2-PERF), finding that the heuristic improvements drive performance while low-level optimizations have mixed impact. The work discusses limitations, reproducibility, and future directions, highlighting opportunities for benchmarks, autonomous LLM agents, and cross-language translations to advance optimization research with LLM assistance.

Abstract

The integration of Large Language Models (LLMs) into optimization has created a powerful synergy, opening exciting research opportunities. This paper investigates how LLMs can enhance existing optimization algorithms. Using their pre-trained knowledge, we demonstrate their ability to propose innovative heuristic variations and implementation strategies. To evaluate this, we applied a non-trivial optimization algorithm, Construct, Merge, Solve and Adapt (CMSA) -- a hybrid metaheuristic for combinatorial optimization problems that incorporates a heuristic in the solution construction phase. Our results show that an alternative heuristic proposed by GPT-4o outperforms the expert-designed heuristic of CMSA, with the performance gap widening on larger and denser graphs. Project URL: https://imp-opt-algo-llms.surge.sh/

Improving Existing Optimization Algorithms with LLMs

TL;DR

The paper investigates whether LLMs can improve existing optimization algorithms by treating LLMs as assistants that suggest novel heuristics and implementation improvements. It uses an expert CMSA implementation for the Maximum Independent Set (MIS) problem as a testbed and demonstrates that an LLM-generated heuristic (LLM-CMSA-V1) outperforms the expert-designed CMSA, with larger gains on bigger, denser graphs. It extends the approach with an entropy-based diversification (LLM-CMSA-V2) and subsequent code-optimization variants (V1-PERF, V2-PERF), finding that the heuristic improvements drive performance while low-level optimizations have mixed impact. The work discusses limitations, reproducibility, and future directions, highlighting opportunities for benchmarks, autonomous LLM agents, and cross-language translations to advance optimization research with LLM assistance.

Abstract

The integration of Large Language Models (LLMs) into optimization has created a powerful synergy, opening exciting research opportunities. This paper investigates how LLMs can enhance existing optimization algorithms. Using their pre-trained knowledge, we demonstrate their ability to propose innovative heuristic variations and implementation strategies. To evaluate this, we applied a non-trivial optimization algorithm, Construct, Merge, Solve and Adapt (CMSA) -- a hybrid metaheuristic for combinatorial optimization problems that incorporates a heuristic in the solution construction phase. Our results show that an alternative heuristic proposed by GPT-4o outperforms the expert-designed heuristic of CMSA, with the performance gap widening on larger and denser graphs. Project URL: https://imp-opt-algo-llms.surge.sh/

Paper Structure

This paper contains 18 sections, 5 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Code Generation with LLMs
  4. Maximum Independent Set (MIS) Problem
  5. CMSA
  6. LLM-Improved CMSA for MIS
  7. Beginning of the story
  8. First Steps with LLMs
  9. Discovering New Heuristics
  10. New heuristic from the LLM
  11. Improving LLM-CMSA-V1 with the LLM
  12. Code Optimization Strategies
  13. Reproducility
  14. Empirical Evaluation
  15. Preliminary
  16. ...and 3 more sections

Figures (6)

  • Figure 1: A dialogue showing how a chatbot applies our approach to improving optimization algorithms.
  • Figure 2: Examples of maximum independent sets.
  • Figure 3: Two LLM interaction patterns: (a) a direct request to improve a heuristic using CMSA's age parameter, and (b) an iterative dialogue to enhance both heuristic quality and C++ performance through error correction. Both use in-context learning as a prompting strategy dong2024surveyincontextlearningLi2023-ij.
  • Figure 4: Comparative analysis of solution quality: Original CMSA vs. LLM-CMSA variants (V1 and V2).
  • Figure 5: Critical difference (CD) plots for all graph types
  • ...and 1 more figures