Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization

Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, Alex Dimakis, Ion Stoica

TL;DR

This work introduces AdaEvolve, a framework that reformulates LLM-driven evolution as a hierarchical adaptive optimization problem and demonstrates that AdaEvolve consistently outperforms the open-sourced baselines across 185 different open-ended optimization problems including combinatorial, systems optimization and algorithm design problems.

Abstract

The paradigm of automated program generation is shifting from one-shot generation to inference-time search, where Large Language Models (LLMs) function as semantic mutation operators within evolutionary loops. While effective, these systems are currently governed by static schedules that fail to account for the non-stationary dynamics of the search process. This rigidity results in substantial computational waste, as resources are indiscriminately allocated to stagnating populations while promising frontiers remain under-exploited. We introduce AdaEvolve, a framework that reformulates LLM-driven evolution as a hierarchical adaptive optimization problem. AdaEvolve uses an "accumulated improvement signal" to unify decisions across three levels: Local Adaptation, which dynamically modulates the exploration intensity within a population of solution candidates; Global Adaptation, which routes the global resource budget via bandit-based scheduling across different solution candidate populations; and Meta-Guidance which generates novel solution tactics based on the previously generated solutions and their corresponding improvements when the progress stalls. We demonstrate that AdaEvolve consistently outperforms the open-sourced baselines across 185 different open-ended optimization problems including combinatorial, systems optimization and algorithm design problems.

AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization

TL;DR

This work introduces AdaEvolve, a framework that reformulates LLM-driven evolution as a hierarchical adaptive optimization problem and demonstrates that AdaEvolve consistently outperforms the open-sourced baselines across 185 different open-ended optimization problems including combinatorial, systems optimization and algorithm design problems.

Abstract

The paradigm of automated program generation is shifting from one-shot generation to inference-time search, where Large Language Models (LLMs) function as semantic mutation operators within evolutionary loops. While effective, these systems are currently governed by static schedules that fail to account for the non-stationary dynamics of the search process. This rigidity results in substantial computational waste, as resources are indiscriminately allocated to stagnating populations while promising frontiers remain under-exploited. We introduce AdaEvolve, a framework that reformulates LLM-driven evolution as a hierarchical adaptive optimization problem. AdaEvolve uses an "accumulated improvement signal" to unify decisions across three levels: Local Adaptation, which dynamically modulates the exploration intensity within a population of solution candidates; Global Adaptation, which routes the global resource budget via bandit-based scheduling across different solution candidate populations; and Meta-Guidance which generates novel solution tactics based on the previously generated solutions and their corresponding improvements when the progress stalls. We demonstrate that AdaEvolve consistently outperforms the open-sourced baselines across 185 different open-ended optimization problems including combinatorial, systems optimization and algorithm design problems.
Paper Structure (38 sections, 6 equations, 5 figures, 8 tables, 6 algorithms)

This paper contains 38 sections, 6 equations, 5 figures, 8 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Works
  4. Test-Time Scaling and Search Algorithms.
  5. LLM-Guided Evolutionary Search.
  6. Adaptive Optimization and Control.
  7. The AdaEvolve Framework
  8. Problem Formulation
  9. Level 1: Local Adaptation (Within Island Exploration Intensity)
  10. The Accumulated Improvement Signal.
  11. Dynamic Exploration Intensity.
  12. Level 2: Global Adaptation (Across Island Resource Allocation)
  13. Decayed-Magnitude Bandit-Based Island Selection.
  14. Migration Across Islands.
  15. Dynamic Island Spawning.
  16. ...and 23 more sections

Figures (5)

  • Figure 1: AdaEvolve overview. Left: Standard LLM-guided search relies on fixed optimization policies, with static schedules, uniform resource allocation, and rigid prompts. Center: AdaEvolve introduces hierarchical adaptivity by dynamically modulating exploration intensity, reallocating compute across populations of programs (islands), and generating meta-level guidance from a unified improvement signal. Right: AdaEvolve overcomes stagnation, reaching to a best-known score of 2.636 on the Circle Packing (N = 26), surpassing the Human SOTA (2.634) and AlphaEvolve (2.635).
  • Figure 2: Comparison of the evolutionary algorithms on Circle Packing (Square $n=26$) and Heilbronn Triangles ($n=11$) problems using GPT-5 backbone for all of them. $n$ is a parameter of the optimization problems we explain in \ref{['tab:adaevolve_problem_defs']}.
  • Figure 3: GPT-5 best configurations comparisons for the Circle Packing experiment.
  • Figure 3: Frontier-CS Benchmark Results
  • Figure 5: AdaEvolve adapts search behavior across tasks. (Left) Signal Processing: exploration transitions to refinement as improvement signals accumulate. (Right) Circle Packing: the evolved strategy breaks stagnation and drives near-optimal layouts.