CCTEST: Testing and Repairing Code Completion Systems

TL;DR

CCTest presents a black-box framework for testing and repairing code completion systems. It employs Program Structure-Consistent (PSC) mutations to generate structure-preserving prompt variants, uses a distance-based metamorphic testing oracle to detect inconsistent outputs, and repairs results by selecting the output most representative of the non-outlier set. Evaluated on eight LLM-based systems with tens of thousands of prompts, it uncovers thousands of defects and yields substantial improvements in BLEU and edit similarity, validated by human judges. The approach is language-agnostic in principle and demonstrates practical potential for improving AI-assisted coding workflows without retraining models.

Abstract

Code completion, a highly valuable topic in the software development domain, has been increasingly promoted for use by recent advances in large language models (LLMs). To date, visible LLM-based code completion frameworks such as GitHub Copilot and GPT are trained using deep learning over vast quantities of unstructured text and open source code. As the paramount component and the cornerstone in daily programming tasks, code completion has largely boosted professionals' efficiency in building real-world software systems. In contrast to this flourishing market, we find that code completion systems often output suspicious results, and to date, an automated testing and enhancement framework for code completion systems is not available. This research proposes CCTEST, a framework to test and repair code completion systems in blackbox settings. CCTEST features a set of novel mutation strategies, namely program structure-correlated (PSC) mutations, to generate mutated code completion inputs. Then, it detects inconsistent outputs, representing possibly erroneous cases, from all the completed code cases. Moreover, CCTEST repairs the code completion outputs by selecting the output that mostly reflects the "average" appearance of all output cases, as the final output of the code completion systems. We detected a total of 33,540 inputs (with a true positive rate of 86%) that can trigger erroneous cases from eight popular LLM-based code completion systems. With repairing, we show that the accuracy of code completion systems is notably increased by 40% and 67% with respect to BLEU score and Levenshtein edit similarity.

Figures (12)

• Figure 1: Motivating example over Codegen (input and autocompletion is marked with gray and white background, respectively).
• Figure 2: A holistic view of LLM-based code completion system and how CCTest facilitates testing and enhancement.
• Figure 3: Workflow of CCTest. CCTest launches PSC testing by mutating a prompt into a collection of prompt variants, cross compares code completion results, and enables automated code completion enhancement.
• Figure 4: Comparing the syntactic form with ground truth appears overly strict and lead to false positives. The code snippets are simplified for readability.
• Figure 5: Outlier case study. The code completion output of the seed prompt is largely deviated, whereas a mutated prompt results in much better code completion output. The "completion result" (code in white) in (a) means ground truth.