Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the Efficacy of Large Language Models for Code Clone Detection

Mohamad Khajezade, Jie JW Wu, Fatemeh Hendijani Fard, Gema Rodríguez-Pérez, Mohamed Sami Shehata

TL;DR

This work investigates the use of Large Language Models for Code Clone Detection (CCD), emphasizing non-generative Type-4 clones. Using CodeNet-derived mono-lingual (Java-Java) and cross-language (Java-Ruby) datasets, the authors design prompts for zero-shot CCD and compare ChatGPT against fully fine-tuned baselines. The results show ChatGPT achieves $F1$ scores of $0.877$ (cross-language) and $0.878$ (mono-lingual), with prompt design and problem difficulty significantly influencing performance. The study highlights the potential of LLMs for CCD tasks and outlines directions for evaluating additional models and languages to broaden applicability and understanding of prompt-based CCD capabilities.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in various natural language processing and software engineering tasks, such as code generation. The LLMs are mainly utilized in the prompt-based zero/few-shot paradigm to guide the model in accomplishing the task. GPT-based models are one of the popular ones studied for tasks such as code comment generation or test generation. These tasks are `generative' tasks. However, there is limited research on the usage of LLMs for `non-generative' tasks such as classification using the prompt-based paradigm. In this preliminary exploratory study, we investigated the applicability of LLMs for Code Clone Detection (CCD), a non-generative task. By building a mono-lingual and cross-lingual CCD dataset derived from CodeNet, we first investigated two different prompts using ChatGPT to detect Type-4 code clones in Java-Java and Java-Ruby pairs in a zero-shot setting. We then conducted an analysis to understand the strengths and weaknesses of ChatGPT in CCD. ChatGPT surpasses the baselines in cross-language CCD attaining an F1-score of 0.877 and achieves comparable performance to fully fine-tuned models for mono-lingual CCD, with an F1-score of 0.878. Also, the prompt and the difficulty level of the problems has an impact on the performance of ChatGPT. Finally we provide insights and future directions based on our initial analysis

Investigating the Efficacy of Large Language Models for Code Clone Detection

TL;DR

This work investigates the use of Large Language Models for Code Clone Detection (CCD), emphasizing non-generative Type-4 clones. Using CodeNet-derived mono-lingual (Java-Java) and cross-language (Java-Ruby) datasets, the authors design prompts for zero-shot CCD and compare ChatGPT against fully fine-tuned baselines. The results show ChatGPT achieves scores of (cross-language) and (mono-lingual), with prompt design and problem difficulty significantly influencing performance. The study highlights the potential of LLMs for CCD tasks and outlines directions for evaluating additional models and languages to broaden applicability and understanding of prompt-based CCD capabilities.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in various natural language processing and software engineering tasks, such as code generation. The LLMs are mainly utilized in the prompt-based zero/few-shot paradigm to guide the model in accomplishing the task. GPT-based models are one of the popular ones studied for tasks such as code comment generation or test generation. These tasks are `generative' tasks. However, there is limited research on the usage of LLMs for `non-generative' tasks such as classification using the prompt-based paradigm. In this preliminary exploratory study, we investigated the applicability of LLMs for Code Clone Detection (CCD), a non-generative task. By building a mono-lingual and cross-lingual CCD dataset derived from CodeNet, we first investigated two different prompts using ChatGPT to detect Type-4 code clones in Java-Java and Java-Ruby pairs in a zero-shot setting. We then conducted an analysis to understand the strengths and weaknesses of ChatGPT in CCD. ChatGPT surpasses the baselines in cross-language CCD attaining an F1-score of 0.877 and achieves comparable performance to fully fine-tuned models for mono-lingual CCD, with an F1-score of 0.878. Also, the prompt and the difficulty level of the problems has an impact on the performance of ChatGPT. Finally we provide insights and future directions based on our initial analysis
Paper Structure (9 sections, 1 equation, 3 figures, 5 tables)

This paper contains 9 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Investigations
  3. Investigations Methodology
  4. Evaluation Metrics
  5. Clone Dataset (Type-4)
  6. Baselines
  7. Investigation Results
  8. Discussions
  9. Conclusions and Future Work

Figures (3)

  • Figure 1: Code1 and code2 are two accepted submissions for problem #5 of CodeNET dataset puri2021project. The figure displays the result of querying ChatGPT using a basic prompt asking whether code1 and code2 are code clones. Though ChatGPT correctly identifies the functionality of both scripts, it fails to identify whether these two are clones.
  • Figure 2: Code1 and code2 are two accepted submissions for problem #5 of CodeNET dataset puri2021project as shown in Figure \ref{['fig:motivation']}. The figure displays the result of querying ChatGPT using our designed prompt when it correctly mentions that the two codes are solving the same problem.
  • Figure 3: a) Comparison of the average Acceptance Rates for positive (blue) and negative (orange) misclassified samples, and the selected CodeNet problems (green). The lower Acceptance Rate indicates more sophisticated problems. b) Comparison of the average Cyclomatic Complexity for positive (blue) misclassified samples and negative (orange) mislabeled samples, and the selected programs from CodeNET (green). A higher CC represents more difficult problems.