Pareto-Grid-Guided Large Language Models for Fast and High-Quality Heuristics Design in Multi-Objective Combinatorial Optimization

TL;DR

The paper tackles multi-objective combinatorial optimization by integrating large language models with a Pareto-grid guided SEMO framework (MPaGE) to automatically design diverse heuristics that balance solution quality and runtime. MPaGE partitions the objective space with a Pareto Front Grid, uses LLMs to review and cluster semantically distinct heuristic behaviors, and guides variation across clusters to foster diversity while maintaining convergence toward the Pareto front. It formalizes Language Multi-Criteria Heuristic Design (LMHD) with a two-objective evaluation (quality and efficiency) and leverages semantic clustering to enable cross-cluster recombination and targeted mutations. Extensive experiments on Bi-TSP, Tri-TSP, Bi-CVRP, and Bi-KP demonstrate that MPaGE yields superior hypervolume and competitive IGD with significantly faster runtimes and higher diversity (SWDI/CDI) than prior LLM-based methods and traditional MOEAs, highlighting its scalability and practical impact for automated heuristic discovery in MOCOP. The work also provides a framework for designing and evaluating LLM-generated heuristics, including prompts, reflection feedback, and clustering strategies, contributing to reproducibility and future research in automated solver synthesis.

Abstract

Multi-objective combinatorial optimization problems (MOCOP) frequently arise in practical applications that require the simultaneous optimization of conflicting objectives. Although traditional evolutionary algorithms can be effective, they typically depend on domain knowledge and repeated parameter tuning, limiting flexibility when applied to unseen MOCOP instances. Recently, integration of Large Language Models (LLMs) into evolutionary computation has opened new avenues for automatic heuristic generation, using their advanced language understanding and code synthesis capabilities. Nevertheless, most existing approaches predominantly focus on single-objective tasks, often neglecting key considerations such as runtime efficiency and heuristic diversity in multi-objective settings. To bridge this gap, we introduce Multi-heuristics for MOCOP via Pareto-Grid-guided Evolution of LLMs (MPaGE), a novel enhancement of the Simple Evolutionary Multiobjective Optimization (SEMO) framework that leverages LLMs and Pareto Front Grid (PFG) technique. By partitioning the objective space into grids and retaining top-performing candidates to guide heuristic generation, MPaGE utilizes LLMs to prioritize heuristics with semantically distinct logical structures during variation, thus promoting diversity and mitigating redundancy within the population. Through extensive evaluations, MPaGE demonstrates superior performance over existing LLM-based frameworks, and achieves competitive results to traditional Multi-objective evolutionary algorithms (MOEAs), with significantly faster runtime. Our code is available at: https://github.com/langkhachhoha/MPaGE.

Figures (8)

• Figure 1: Overview of the proposed method. The heuristics are first represented in a feature space, and then partitioned into grid cells based on their objective values. Potential parent heuristics are then chosen with the assistance of LLM-based review from grids. Crossover and mutation operations are applied, guided by informative feedback from LLM-based reflection to enhance population diversity. Finally, non-dominated individuals are retained to form the next generation for the subsequent iteration.
• Figure 2: LLM-based review clustering applies elitism heuristics based on semantic similarity and employs a probability matrix to guide variation among clusters.
• Figure 3: Group Heuristics 1: Three heuristics implementing the same logic: selecting a solution and generating a neighbor by randomly swapping two elements.
• Figure 4: Group Heuristics 2: Three heuristics implementing the same logic: selecting a solution and reversing a randomly chosen segment $[i:j+1]$.
• Figure 5: Group Heuristics 3: Four heuristics implementing the same logic: Sequentially select unvisited nodes such that the next node in the closet with the smallest sum of total distance.