How the Misuse of a Dataset Harmed Semantic Clone Detection
Jens Krinke, Chaiyong Ragkhitwetsagul
TL;DR
The paper investigates BigCloneBench and its suitability as ground truth for semantic clone detection. Through a large-scale manual review of WT3/T4 pairs and a literature survey of papers using BigCloneBench, the authors demonstrate pervasive mislabelling and bias in the ground truth, with 93% of WT3/T4 pairs failing to share the intended functionality. They show that many reported high F1 scores in semantic cloning studies likely reflect dataset artefacts rather than genuine semantic similarity, and they reveal substantial cross-functional and intra-functional labeling issues as well as imbalances across functionalities. The work discusses alternative benchmarks, reports attempts at repairing the dataset, and argues for careful, critical use of BigCloneBench for traditional clone detection while cautioning against relying on it for machine learning in semantic clone detection. The authors provide data and protocols to enable replication and call for improved ground-truth construction in future benchmarks to safeguard the validity and generalizability of results in semantic code similarity research.
Abstract
BigCloneBench is a well-known and widely used large-scale dataset for the evaluation of recall of clone detection tools. It has been beneficial for research on clone detection and has become a standard in evaluating the performance of clone detection tools. More recently, it has also been widely used as a dataset to evaluate machine learning approaches to semantic clone detection or code similarity detection for functional or semantic similarity. This paper demonstrates that BigCloneBench is problematic to use as ground truth for learning or evaluating semantic code similarity, and highlights the aspects of BigCloneBench that affect the ground truth quality. A manual investigation of a statistically significant random sample of 406 Weak Type-3/Type-4 clone pairs revealed that 93% of them do not have a similar functionality and are therefore mislabelled. In a literature review of 179 papers that use BigCloneBench as a dataset, we found 139 papers that used BigCloneBench to evaluate semantic clone detection and where the results are threatened in their validity by the mislabelling. As such, these papers often report high F1 scores (e.g., above 0.9), which indicates overfitting to dataset-specific artefacts rather than genuine semantic similarity detection. We emphasise that using BigCloneBench remains valid for the intended purpose of evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones. We acknowledge the important contributions of BigCloneBench to two decades of traditional clone detection research. However, the usage of BigCloneBench beyond the intended purpose without careful consideration of its limitations has led to misleading results and conclusions, and potentially harmed the field of semantic clone detection.