The Art of Being Difficult: Combining Human and AI Strengths to Find Adversarial Instances for Heuristics

TL;DR

The paper introduces Co-FunSearch, a framework for human–LLM collaboration to generate adversarial instances that reveal weaknesses in standard heuristics for NP-hard problems. By iteratively refining LLM-generated programs and extracting structural insights, the authors obtain state-of-the-art lower bounds across four problems: the Nemhauser–Ullmann knapsack heuristic, Best-Fit for bin packing, hierarchical $k$-median clustering, and Lovász’s gasoline problem. The results include disproving that NU is output-polynomial, improving the random-order lower bound for Best-Fit to $1.5$, proving a golden-ratio lower bound for the price of hierarchy in $k$-median, and producing counterexamples to a 2-approximation conjecture in the gasoline problem. Overall, the work demonstrates that expert oversight can extrapolate algorithmic insights from LLM-driven search to break long-standing barriers, highlighting LLM-assisted collaboration as a valuable tool in mathematical and CS research.

Abstract

We demonstrate the power of human-LLM collaboration in tackling open problems in theoretical computer science. Focusing on combinatorial optimization, we refine outputs from the FunSearch algorithm [Romera-Paredes et al., Nature 2023] to derive state-of-the-art lower bounds for standard heuristics. Specifically, we target the generation of adversarial instances where these heuristics perform poorly. By iterating on FunSearch's outputs, we identify improved constructions for hierarchical $k$-median clustering, bin packing, the knapsack problem, and a generalization of Lovász's gasoline problem - some of these have not seen much improvement for over a decade, despite intermittent attention. These results illustrate how expert oversight can effectively extrapolate algorithmic insights from LLM-based evolutionary methods to break long-standing barriers. Our findings demonstrate that while LLMs provide critical initial patterns, human expertise is essential for transforming these patterns into mathematically rigorous and insightful constructions. This work highlights that LLMs are a strong collaborative tool in mathematics and computer science research.

Figures (6)

• Figure 1: A diagrammatic representation of Co-FunSearch.
• Figure 2: The evolution of programs generating bin packing instances, with model open-mistral-nemo and a temperature of $1.5$.
• Figure 3: Comparing the effect of different hyperparameters on the objective function in bin packing.(a) Comparing different models, each with temperature 1.0 and starting with a hard-coded instance.(b) Comparing rolling average (10 samples) and max-performance of gpt-4.1-mini with gpt-4.1-nano, with temp: $1.0$.(c) Variation of different sampling temperatures for gpt-4.1-mini, each starting with a hard-coded instance.(d) Variation of initial instances for gpt-4.1-mini with temperature $1.0$.
• Figure 4: The evolution of programs generating instances of the knapsack problem. The model used was gpt-4.1-nano with a temperature of $1.0$, and results obtainable despite a bug in the implementation that underestimated the sizes of some Pareto sets.
• Figure 5: The evolution of programs generating clustering-instances. The model used was open-mistral-nemo with a temperature of $1.5$.

Theorems & Definitions (10)

• Theorem 3.1
• Theorem 3.2
• Theorem 3.3
• proof
• Lemma 5.1
• proof
• Lemma 5.2
• proof
• Lemma 5.3
• proof