Industrial-Scale Neural Network Clone Detection with Disk-Based Similarity Search

TL;DR

DB-SSCD addresses industrial-scale neural code clone detection by moving embedding search from RAM to disk, enabling indexing and querying over corpora larger than memory. It builds on SSCD by persisting embeddings and indices on disk using Faiss and Milvus DiskANN, and adds a Type-1 deduplication strategy and a Global Top-K ranking to deliver consistent results across batches. Experiments on BigCloneBench and The Stack show disk-based recall close to in-memory approaches, with about a $2\times$ slowdown due to disk I/O, validating scalability to over a billion lines of code. The work highlights practical trade-offs between indexing speed, query throughput, and storage footprint, and suggests future directions such as sub-function-level clone detection and enhanced optimization.

Abstract

Code clones are similar code fragments that often arise from copy-and-paste programming. Neural networks can classify pairs of code fragments as clone/not-clone with high accuracy. However, finding clones in industrial-scale code needs a more scalable approach than pairwise comparison. We extend existing neural network-based clone detection schemes to handle codebases that far exceed available memory, using indexing and search methods for external storage such as disks and solid-state drives. We generate a high-dimensional vector embedding for each code fragment using a transformer-based neural network. We then find similar embeddings using efficient multidimensional nearest neighbor search algorithms on external storage to find similar embeddings without pairwise comparison. We identify specific problems with industrial-scale code bases, such as large sets of almost identical code fragments that interact poorly with $k$-nearest neighbour search algorithms, and provide an effective solution. We demonstrate that our disk-based clone search approach achieves similar clone detection accuracy as an equivalent in-memory technique. Using a solid-state drive as external storage, our approach is around 2$\times$ slower than the in-memory approach for a problem size that can fit within memory. We further demonstrate that our approach can scale to over a billion lines of code, providing valuable insights into the trade-offs between indexing speed, query performance, and storage efficiency for industrial-scale code clone detection.

Figures (3)

• Figure 1: DB-SSCD: Overall System Architecture for Clone Search
• Figure 2: Global Top-K Strategy: Recall and Precision Trade-off for Global Top-K on the Company-C Dataset.
• Figure 3: Illustration of the Impact of Type-1 Clone Deduplication on Top-k Candidate Diversity