An Empirical Study of LLM-Based Code Clone Detection

TL;DR

This study interrogates the generalization and response stability of LLMs for code clone detection. By constructing seven CN/BCB-derived datasets and evaluating five LLMs across four prompts, it shows that CN-derived data yield higher $F1$ scores (e.g., up to 0.943) while BCB-derived data degrade performance, underscoring cross-dataset generalization challenges. It also reveals that most models maintain high response consistency (>90%), with prompt choice often exerting a larger influence than temperature on both stability and $F1$, though some models (e.g., Llama 3.1) are more variable. The findings suggest that although LLMs can perform strongly on CN-type data, fine-tuning and careful prompt design are needed to achieve reliable clone detection on diverse, real-world codebases; the authors provide datasets and scripts to support reproducibility.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in various software engineering tasks, such as code generation and debugging, because of their ability to translate between programming languages and natural languages. Existing studies have demonstrated the effectiveness of LLMs in code clone detection. However, two crucial issues remain unaddressed: the ability of LLMs to achieve comparable performance across different datasets and the consistency of LLMs' responses in code clone detection. To address these issues, we constructed seven code clone datasets and then evaluated five LLMs in four existing prompts with these datasets. The datasets were created by sampling code pairs using their Levenshtein ratio from two different code collections, CodeNet and BigCloneBench. Our evaluation revealed that although LLMs perform well in CodeNet-related datasets, with o3-mini achieving a 0.943 F1 score, their performance significantly decreased in BigCloneBench-related datasets. Most models achieved a high response consistency, with over 90\% of judgments remaining consistent across all five submissions. The fluctuations of the F1 score affected by inconsistency are also tiny; their variations are less than 0.03.

Figures (4)

• Figure 1: Experiment Process
• Figure 2: F1-Score of Each Model and Prompt for Different Datasets
• Figure 3: Recall, Precision, and F1-Score of Each Model and Prompt in Each Similarity Range
• Figure 4: Response Consistency Rate (RCR) of Each Model