Selecting and Combining Large Language Models for Scalable Code Clone Detection

TL;DR

The paper tackles scalable code clone detection by systematically evaluating 9 novel LLMs across diverse public and private datasets, revealing strong dataset dependence in model performance. It introduces a rigorous LLM-identification pipeline and a unified evaluation framework, including a Borda-count aggregation to compare models comprehensively. Beyond individual model performance, it investigates ensemble strategies, showing that normalization and aggregation choices critically influence gains, with notable precision improvements on large in-situ datasets. The findings emphasize data quality, tokenizer configuration, and domain-aligned training data over mere model scaling, offering practical guidance for deploying LLM-based clone detection at scale and suggesting avenues for multilingual expansion and more sophisticated ensemble methods.

Abstract

Source code clones pose risks ranging from intellectual property violations to unintended vulnerabilities. Effective and efficient scalable clone detection, especially for diverged clones, remains challenging. Large language models (LLMs) have recently been applied to clone detection tasks. However, the rapid emergence of LLMs raises questions about optimal model selection and potential LLM-ensemble efficacy. This paper addresses the first question by identifying 76 LLMs and filtering them down to suitable candidates for large-scale clone detection. The candidates were evaluated on two public industrial datasets, BigCloneBench, and a commercial large-scale dataset. No uniformly 'best-LLM' emerged, though CodeT5+110M, CuBERT and SPTCode were top-performers. Analysis of LLM-candidates suggested that smaller embedding sizes, smaller tokenizer vocabularies and tailored datasets are advantageous. On commercial large-scale dataset a top-performing CodeT5+110M achieved 39.71\% precision: twice the precision of previously used CodeBERT. To address the second question, this paper explores ensembling of the selected LLMs: effort-effective approach to improving effectiveness. Results suggest the importance of score normalization and favoring ensembling methods like maximum or sum over averaging. Also, findings indicate that ensembling approach can be statistically significant and effective on larger datasets: the best-performing ensemble achieved even higher precision of 46.91\% over individual LLM on the commercial large-scale code.

Figures (5)

• Figure 1: Illustrative example: code inference with CodeBERT LLM
• Figure 2: Diagram of SSCD
• Figure 3: LLM selection process
• Figure 4: Avg. Recall by Normalization & Aggregation for Company-C
• Figure 5: Avg. Recall by Normalization & Aggregation for Company-C++