Rethinking Code Similarity for Automated Algorithm Design with LLMs

TL;DR

BehaveSim is proposed, a novel method to measure algorithmic similarity through the lens of problem-solving behavior as a sequence of intermediate solutions produced during execution, dubbed as problem-solving trajectories (PSTrajs).

Abstract

The rise of Large Language Model-based Automated Algorithm Design (LLM-AAD) has transformed algorithm development by autonomously generating code implementations of expert-level algorithms. Unlike traditional expert-driven algorithm development, in the LLM-AAD paradigm, the main design principle behind an algorithm is often implicitly embedded in the generated code. Therefore, assessing algorithmic similarity directly from code, distinguishing genuine algorithmic innovation from mere syntactic variation, becomes essential. While various code similarity metrics exist, they fail to capture algorithmic similarity, as they focus on surface-level syntax or output equivalence rather than the underlying algorithmic logic. We propose BehaveSim, a novel method to measure algorithmic similarity through the lens of problem-solving behavior as a sequence of intermediate solutions produced during execution, dubbed as problem-solving trajectories (PSTrajs). By quantifying the alignment between PSTrajs using dynamic time warping (DTW), BehaveSim distinguishes algorithms with divergent logic despite syntactic or output-level similarities. We demonstrate its utility in two key applications: (i) Enhancing LLM-AAD: Integrating BehaveSim into existing LLM-AAD frameworks (e.g., FunSearch, EoH) promotes behavioral diversity, significantly improving performance on three AAD tasks. (ii) Algorithm analysis: BehaveSim clusters generated algorithms by behavior, enabling systematic analysis of problem-solving strategies--a crucial tool for the growing ecosystem of AI-generated algorithms. Data and code of this work are open-sourced at https://github.com/RayZhhh/behavesim.

Figures (11)

• Figure 1: Examples demonstrating existing code similarity metrics are insufficient for measuring algorithmic similarity. (a) Existing code similarity metrics, on the one hand, find the breadth-first search (BFS) and depth-first search (DFS) algorithms highly similar, despite the two algorithms being inherently different in their traversal logic. On the other hand, they also find that a re-implementation of DFS (Code 3) based on recursion is a completely different algorithm from the original iteration-based implementation (Code 2), despite the two codes essentially representing the same algorithm. (b) Merely checking the output of two algorithms cannot distinguish between two distinct algorithms. Both insertion sort and bubble sort algorithms yield identical sorted arrays, despite being inherently different algorithms.
• Figure 2: Problem-solving behaviors on the traveling salesman problem (TSP) for two algorithms with highly similar codes. The only distinction in their implementations lies in the use of argmin() and argmax(), which leads to profoundly different behaviors: Algorithm 1 chooses the nearest neighbor node of the current node, while Algorithm 2 always steps to the farthest node away from the current node. Nevertheless, existing similarity metrics still assign them a high degree of similarity, failing to reveal their behavioral differences.
• Figure 3: (a) Comparison of the problem-solving behavior for optimization algorithms. Each plot shows the trajectories of two algorithms on the Rosenbrock problem, each initialized at two different locations. Behavioral similarity is reported using both mean pairwise distance and DTW. (b) Comparison of TSP algorithms. Each plot shows the partial solution at different steps (5, 25, 50). The red star denotes the starting city.
• Figure 4: Convergence of the top-10 algorithm sets produced by different search methods on three AAD tasks. The y-axis reports the relative gap to the best-known optimum (lower is better). Markers denote the mean over three independent runs; shaded regions indicate standard deviation.
• Figure 5: Clustering results based on two similarity measures. (i) Code 11 and Code 12 are clustered together by CodeBLEU, yet they exhibit distinct behaviors due to differences in their logic for computing the final score. (ii) Code 6 and Code 8 differ in code structure but display consistent behaviors. This is because a bug in Code 6 sets the "future_potentials" field to an all-zero array, making its score solely determined by the distance to neighbors. Similarly, Code 8 assigns a dominant weight to the distance term, which overrides other factors. As a result, both Code 6 and Code 8 essentially follow the strategy of selecting the nearest unvisited neighbor.