Rethinking LLM-Driven Heuristic Design: Generating Efficient and Specialized Solvers via Dynamics-Aware Optimization

TL;DR

This work reframes solver design as a time-evolving dynamical process, introducing a trajectory-aware metric, tLDR, to quantify convergence speed and stability. It proposes DASH, a framework that co-evolves solver mechanisms and runtime schedules via three layers (MDL, MCL, SSL) under a convergence-aware acceptance protocol, and pairs it with Profiled Library Retrieval (PLR) to enable cost-efficient, distribution-aware warm starts. Empirically, DASH delivers over 3x runtime improvements and superior solution quality across four combinatorial problems, while maintaining accuracy under distribution shifts and reducing LLM adaptation costs by more than 90% through PLR. The approach demonstrates strong generalization across solver backbones (GLS/ILS/LKH) and instance distributions, suggesting practical impact for deploying efficient, specialized solvers in diverse, real-world settings. Together, the trajectory-centric perspective and PLR enable robust, scalable LLM-driven heuristic design with practical performance gains.

Abstract

Large Language Models (LLMs) have advanced the field of Combinatorial Optimization through automated heuristic generation. Instead of relying on manual design, this LLM-Driven Heuristic Design (LHD) process leverages LLMs to iteratively generate and refine solvers to achieve high performance. However, existing LHD frameworks face two critical limitations: (1) Endpoint-only evaluation, which ranks solvers solely by final quality, ignoring the convergence process and runtime efficiency; (2) High adaptation costs, where distribution shifts necessitate re-adaptation to generate specialized solvers for new instance groups. To address these issues, we propose Dynamics-Aware Solver Heuristics (DASH), a framework that co-optimizes solver search mechanisms and runtime schedules guided by a convergence-aware metric, thereby identifying efficient and high-performance solvers. Furthermore, to mitigate expensive re-adaptation, DASH incorporates Profiled Library Retrieval (PLR). PLR efficiently archives specialized solvers concurrently with the evolutionary process, enabling cost-effective warm-starts for heterogeneous distributions. Experiments on four combinatorial optimization problems demonstrate that DASH improves runtime efficiency by over 3 times, while surpassing the solution quality of state-of-the-art baselines across diverse problem scales. Furthermore, by enabling profile-based warm starts, DASH maintains superior accuracy under different distributions while cutting LLM adaptation costs by over 90%.

Figures (7)

• Figure 1: Motivating Experiments on TSP tasks using Guided Local Search voudouris1999gls as solver backbone and under a 10s time limit. (Left) Three solvers reach similar final gaps with different trajectories, rendering them indistinguishable under endpoint-only evaluation. Solver 1 converges earlier, achieving lower gaps at earlier times and leaving more of the time budget for subsequent improvements. (Right) Performance significantly degrades when solvers are transferred across shifts in problem size (node count) or density (node clustering), underscoring the necessity of generating specialized solvers for distinct instance distributions.
• Figure 2: Overview of the DASH framework. Offline evolution co-evolves the solver across MDL, MCL, and SSL using terminal quality and trajectory efficiency for selection. In parallel, PLR maintains group-wise archives from evaluated candidates. At test time, PLR retrieves a group-specific solver to warm-start evalution.
• Figure 3: Evolutionary dynamics on TSP100. Across 5 independent runs (100 evaluations each), we track the best-so-far gap (left) and the corresponding runtime (middle) and tLDR (right). Lines show the mean and shaded bands show the variability across runs. Higher tLDR indicates more efficient anytime improvement.
• Figure 4: Base-model sensitivity on TSP100. Using five base LLMs for solver generation (5 runs, 100 evaluations each), we report the final best-so-far gap (top) and the runtime of that best solver (bottom).
• Figure 5: Instance profiling and grouping (illustration on synthetic TSP instances). We compute a profile vector for each instance (scale, distance statistics, spatial clustering, and shape descriptors), standardize the features, and apply k-means with $G{=}10$. Left: PCA projection of standardized profiles, colored by group assignment. Right: scale--density view using $\log(n)$ and a nearest-neighbor based density proxy.