On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot

TL;DR

This study investigates how semantically equivalent natural language descriptions influence GitHub Copilot's code generation. By generating and evaluating paraphrases for 892 Java methods, the authors show that input wording can alter Copilot predictions in about 46% of cases and affect correctness in roughly 28% of cases, raising questions about robustness and usability of DL-based code generators. They validate paraphrase techniques, provide a detailed replication package, and discuss implications for evaluating and deploying AI pair-programming tools. The work highlights the critical role of input description quality and sets the stage for in vivo studies and improved paraphrasing strategies to enhance reliability.

Abstract

Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate 892 Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in ~46% of cases. Also, differences in the semantically equivalent descriptions might impact the correctness of the generated code ~28%.

Figures (6)

• Figure 1: GitHub Copilot's input for both code context representations
• Figure 2: Results achieved by Copilot when considering the Full context code representation on paraphrases$_{\mathit{manual}}$.
• Figure 3: Results achieved by Copilot when considering the Full context code representation on paraphrases$_{\mathit{PEGASUS}}$ and paraphrases$_{\mathit{TP}}$.
• Figure 4: Example of recommended method that passes the unit tests but reports a low CodeBLEU score compared to the oracle (i.e., target method).
• Figure 5: Example of recommended methods that pass the unit tests but would require 165 edit actions to match the target method.