Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Code Clone Detection Capability of Large Language Models

Zixian Zhang, Takfarinas Saber

TL;DR

This paper investigates code clone detection using GPT-3.5 and GPT-4 across clone types and similarity levels, employing BigCloneBench (human-made) and GPTCloneBench (LLM-generated). It utilizes careful data selection (~300 samples per clone type) and few-shot prompting to compare performance, including both semantic and cross-language clones. Key findings show GPT-4 consistently outperforms GPT-3.5, though both struggle with Type-4 clones, and models exhibit higher accuracy on LLM-generated clones than on human-made ones, indicating potential bias toward familiar patterns. The work highlights the need for broader, multilingual benchmarking and suggests future comparisons with traditional clone-detection tools to assess practical utility.

Abstract

This study aims to assess the performance of two advanced Large Language Models (LLMs), GPT-3.5 and GPT-4, in the task of code clone detection. The evaluation involves testing the models on a variety of code pairs of different clone types and levels of similarity, sourced from two datasets: BigCloneBench (human-made) and GPTCloneBench (LLM-generated). Findings from the study indicate that GPT-4 consistently surpasses GPT-3.5 across all clone types. A correlation was observed between the GPTs' accuracy at identifying code clones and code similarity, with both GPT models exhibiting low effectiveness in detecting the most complex Type-4 code clones. Additionally, GPT models demonstrate a higher performance identifying code clones in LLM-generated code compared to humans-generated code. However, they do not reach impressive accuracy. These results emphasize the imperative for ongoing enhancements in LLM capabilities, particularly in the recognition of code clones and in mitigating their predisposition towards self-generated code clones--which is likely to become an issue as software engineers are more numerous to leverage LLM-enabled code generation and code refactoring tools.

Assessing the Code Clone Detection Capability of Large Language Models

TL;DR

This paper investigates code clone detection using GPT-3.5 and GPT-4 across clone types and similarity levels, employing BigCloneBench (human-made) and GPTCloneBench (LLM-generated). It utilizes careful data selection (~300 samples per clone type) and few-shot prompting to compare performance, including both semantic and cross-language clones. Key findings show GPT-4 consistently outperforms GPT-3.5, though both struggle with Type-4 clones, and models exhibit higher accuracy on LLM-generated clones than on human-made ones, indicating potential bias toward familiar patterns. The work highlights the need for broader, multilingual benchmarking and suggests future comparisons with traditional clone-detection tools to assess practical utility.

Abstract

This study aims to assess the performance of two advanced Large Language Models (LLMs), GPT-3.5 and GPT-4, in the task of code clone detection. The evaluation involves testing the models on a variety of code pairs of different clone types and levels of similarity, sourced from two datasets: BigCloneBench (human-made) and GPTCloneBench (LLM-generated). Findings from the study indicate that GPT-4 consistently surpasses GPT-3.5 across all clone types. A correlation was observed between the GPTs' accuracy at identifying code clones and code similarity, with both GPT models exhibiting low effectiveness in detecting the most complex Type-4 code clones. Additionally, GPT models demonstrate a higher performance identifying code clones in LLM-generated code compared to humans-generated code. However, they do not reach impressive accuracy. These results emphasize the imperative for ongoing enhancements in LLM capabilities, particularly in the recognition of code clones and in mitigating their predisposition towards self-generated code clones--which is likely to become an issue as software engineers are more numerous to leverage LLM-enabled code generation and code refactoring tools.
Paper Structure (17 sections, 4 figures, 3 tables)

This paper contains 17 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Code Clones
  4. Generative Pretrained Transformers (GPTs)
  5. Approach
  6. Datasets
  7. BigCloneBench 10.1109/icsme.2014.77
  8. GPTCloneBench 10.48550/arxiv.2308.13963
  9. Data Selection
  10. Selected Data Profile
  11. GPT Prompt Engineering
  12. Impractical One-Shot Prompt
  13. Selecting Few-Shot Prompt
  14. Results
  15. LLM Performance at Detecting Clone Clones
  16. ...and 2 more sections

Figures (4)

  • Figure 1: BCB Similarity Distribution per Clone Type
  • Figure 2: BCB and GPTCloneBench Code Size Distributions
  • Figure 3: Few-Shot GPT Prompt
  • Figure 4: Performance of LLM models per Code Pair Similarity