Contrastive Concept-Tree Search for LLM-Assisted Algorithm Discovery

TL;DR

The paper tackles the challenge of exploiting the latent semantic structure of the algorithm space in LLM-assisted discovery by introducing Contrastive Concept-Tree Search (CCTS). CCTS builds a hierarchical concept tree from generated programs and uses a contrastive likelihood-ratio score $w(b)=\log\hat{p}_{\eta^{+}}(b)-\log\hat{p}_{\eta^{-}}(b)$ to bias parent selection toward concept combinations associated with improvement, while exploring new concepts via novelty bias. Across Erdős-style tasks and a synthetic environment, CCTS yields faster, more sample-efficient improvements and produces interpretable concept trees; analysis reveals gains largely stem from learning which concepts to avoid. This approach improves search transparency and scalability in algorithm discovery, with implications for handling harder problems and enabling more controllable automated reasoning.

Abstract

Large language Model (LLM)-assisted algorithm discovery is an iterative, black-box optimization process over programs to approximatively solve a target task, where an LLM proposes candidate programs and an external evaluator provides task feedback. Despite intense recent research on the topic and promising results, how can the LLM internal representation of the space of possible programs be maximally exploited to improve performance is an open question. Here, we introduce Contrastive Concept-Tree Search (CCTS), which extracts a hierarchical concept representation from the generated programs and learns a contrastive concept model that guides parent selection. By reweighting parents using a likelihood-ratio score between high- and low-performing solutions, CCTS biases search toward useful concept combinations and away from misleading ones, providing guidance through an explicit concept hierarchy rather than the algorithm lineage constructed by the LLM. We show that CCTS improves search efficiency over fitness-based baselines and produces interpretable, task-specific concept trees across a benchmark of open Erdős-type combinatorics problems. Our analysis indicates that the gains are driven largely by learning which concepts to avoid. We further validate these findings in a controlled synthetic algorithm-discovery environment, which reproduces qualitatively the search dynamics observed with the LLMs.

Figures (17)

• Figure 1: (top) Overview of our algorithm-discovery loop: given a task and evaluator, we repeatedly sample a parent program from an archive, prompt the LLM to generate a mutated child, and evaluate it. The child’s outcome updates both the prompt context and a growing tree of semantic concepts, which then biases parent selection in subsequent iterations. (bottom) Repeating this process yields a phylogenetic program lineage $(x_0,x_1,\ldots)$ and an induced concept tree $(b_0,b_1,\ldots)$. We fit concept utility models on high- vs. low-performing programs via a cross-entropy update, and use their likelihood ratio to guide parent sampling.
• Figure 2: Schematic of the LLM-assisted evolutionary search loop. The red blocks implement the parent selection process, the grey blocks implement the child generation, and the green blocks describe the prompt update process. The blue blocks describe the main novelty of this work: a contrastive exploration process in context space that is used to inform the parent generation process. Boxes containing operators and objects are denoted by dark and light colors, respectively. The notation appearing in the blocks is defined in Sec. \ref{['sec:method']} and Sec. \ref{['sec:CCTS']}.
• Figure 3: Schematic description of the stochastic parent-selection policy mixing uniform exploration and exploitation. At each step, a parent can be selected either uniformly from the archive, with exploration probability $p_\text{explore}$ (exploration), or it can be selected according to a given performance-informed strategy with probability $p_\text{exploit}=1-p_\text{explore}$. Segment sizes are schematic.
• Figure 4: Best score vs. iteration (number of LLM calls) for the circle packing task. Ablations are described in the text. For all tasks, the probability of exploitation is set to $p_{\text{exploit}} = 0.85$. (a) Ran with gemini-flash-2.0 and averaged over 60 runs. (b) Synthetic task averaged over 500 runs. Shaded area show the 95% confidence interval.
• Figure 5: Performance of CCTS compared to baseline methods across multiple tasks. Violin plots show the distribution of the best score after 25 iterations, aggregated over 60 runs; the mean and 95% confidence interval are superimposed. All tasks are run with gemini-flash-2.0 and $p_{\text{exploit}} = 0.85$. The distribution of best scores for CCTS presents thinner tails at low values of final best scores.