Using Code Generation to Solve Open Instances of Combinatorial Design Problems

TL;DR

The paper tackles existence problems for small combinatorial designs by coupling Large Language Models with a verifier-driven code-generation protocol (CPro1) that crafts and tests diverse heuristic constructions in C. By exploiting an oracle verifier and automated hyperparameter tuning, the method explores SA, GA, DFS, and related strategies, successfully solving open instances across six design families, including $PA(N,k,v)$ with $N=21$, $k=14$, $v=9$. The key contributions are an end-to-end workflow for automated design construction, empirical insights into effective strategies (notably constant-temperature SA and GA variants), and a demonstration that LLM-guided experimentation can advance mathematical existence questions. The practical impact lies in scalable, reproducible automation for constructing verifiable combinatorial objects and potential application to other domains where constructive existence can be fully verified.

Abstract

The Handbook of Combinatorial Designs catalogs many types of combinatorial designs, together with lists of open instances for which existence has not yet been determined. We develop a constructive protocol CPro1, which uses Large Language Models (LLMs) to generate code that constructs combinatorial designs and resolves some of these open instances. The protocol starts from a definition of a particular type of design, and a verifier that reliably confirms whether a proposed design is valid. The LLM selects strategies and implements them in code, and scaffolding provides automated hyperparameter tuning and execution feedback using the verifier. Most generated code fails, but by generating many candidates, the protocol automates exploration of a variety of standard methods (e.g. simulated annealing, genetic algorithms) and experimentation with variations (e.g. cost functions) to find successful approaches. Testing on 16 different types of designs, CPro1 constructs solutions to open instances for 6 of them: Symmetric and Skew Weighing Matrices, Equidistant Permutation Arrays, Packing Arrays, Balanced Ternary Designs, and Florentine Rectangles.

Figures (28)

• Figure 1: Constructive Protocol CPro1
• Figure 2: Packing Array example with N=4 k=6 v=3
• Figure 3: Strategies prompt to the LLM, instantiated (italicized text) for Packing Arrays.
• Figure 4: Details prompt to the LLM, instantiated (italicized text) for Packing Arrays with a Simulated Annealing strategy resulting from the prompt in Fig. \ref{['fig:strategiesprompt']}. Note "Do not terminate until a valid solution is found" is not italicized and is a fixed part of the template that is always appended to the strategy description generated by the LLM.
• Figure 5: Code generation prompt to the LLM, instantiated (italicized text) for Packing Arrays. Note this continues the chat following on from the Details prompt in Fig. \ref{['fig:detailsprompt']}; so the Details prompt and response are in context when the LLM responds to this prompt.