NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations

TL;DR

This work tackles the problem of code LLM robustness to natural language prompt variations that occur in real-world coding tasks. It presents a four-stage methodology: deriving NL perturbation categories from literature and practice, validating them via a practitioner survey, automatically composing perturbations with the NLPerturbator, and evaluating six open-source code LLMs on the HumanEval and MBPP benchmarks. The study delivers 18 final perturbation categories (plus 3 co-occurrence combinations) and manually verified datasets HumanEval-R and MBPP-R, revealing that perturbed prompts can significantly degrade code-generation performance, with effects varying by perturbation type and model. These findings underscore the importance of robust prompt engineering and model improvements to handle realistic NL variations in software engineering tasks. The work provides practical implications for prompt design, data collection, and robustness evaluation in code generation, and suggests directions for mitigating sensitivity to NL perturbations.

Abstract

Large language models (LLMs) achieve promising results in code generation based on a given natural language description. They have been integrated into open-source projects and commercial products to facilitate daily coding activities. The natural language description in the prompt is crucial for LLMs to comprehend users' requirements. Prior studies uncover that LLMs are sensitive to the changes in the prompts, including slight changes that look inconspicuous. However, the natural language descriptions often vary in real-world scenarios (e.g., different formats, grammar, and wording). Prior studies on the robustness of LLMs are often based on random perturbations and such perturbations may not actually happen. In this paper, we conduct a comprehensive study to investigate how are code LLMs robust to variations of natural language description in real-world scenarios. We summarize 18 categories of perturbations of natural language and 3 combinations of co-occurred categories based on our literature review and an online survey with practitioners. We propose an automated framework, NLPerturbator, which can perform perturbations of each category given a set of prompts. Through a series of experiments on code generation using six code LLMs, we find that the perturbed prompts can decrease the performance of code generation by a considerable margin (e.g., up to 21.2%, and 4.8% to 6.1% on average). Our study highlights the importance of enhancing the robustness of LLMs to real-world variations in the prompts, as well as the essentiality of attentively constructing the prompts.

Figures (5)

• Figure 1: Example of code generation using original and perturbed prompts. StarCoder starcoder fails to generate a correct solution (i.e., neglects the requirement of letter case) when an extra word "it" is repeated in the natural language description of the problem HumanEval/16.
• Figure 2: Overview of our study.
• Figure 3: Survey participants' rating for each category (RQ1). Categories in the final list are marked in bold; the majority answer is marked with its number of percentage.
• Figure 4: Examples of effectiveness and naturalness in the perturbation (RQ2).
• Figure 5: Examples of code generation using our manually-verified perturbation datasets (RQ3). We only present the part with differences inside each pair of prompts and generated code for brevity.