Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multi-Language Perspective on the Robustness of LLM Code Generation

Fazle Rabbi, Zishuo Ding, Jinqiu Yang

TL;DR

This work investigates the robustness of large language models in code generation across Java, C++, and JavaScript by extending the ReCode framework with a multilingual EvalPlus-X benchmark. It introduces 29 semantic-preserving perturbations across docstrings, function names, syntax, and formatting, and evaluates six LLMs in a black-box setting, complemented by a docstring-repair mitigation attempt. The study reveals language-specific robustness patterns, showing Java generally more resilient while C++ is more fragile, and demonstrates that model size alone does not guarantee robustness. It also shows that prompt repair offers only modest gains and is not a reliable universal solution, underscoring the need for multilingual robustness benchmarks and targeted training strategies, with future work pointing toward white-box analyses and broader language coverage.

Abstract

Large language models have gained significant traction and popularity in recent times, extending their usage to code-generation tasks. While this field has garnered considerable attention, the exploration of testing and evaluating the robustness of code generation models remains an ongoing endeavor. Previous studies have primarily focused on code generation models specifically for the Python language, overlooking other widely-used programming languages. In this work, we conduct a comprehensive comparative analysis to assess the robustness performance of several prominent code generation models and investigate whether robustness can be improved by repairing perturbed docstrings using an LLM. Furthermore, we investigate how their performance varies across different programming languages. To accomplish this, we introduce perturbations in four key areas of the prompt: DocString, functionname, syntax, and format. We have compiled and released a dedicated dataset for this purpose. This work presents our experimental findings, shedding light on the performance of code generation models in various scenarios.

A Multi-Language Perspective on the Robustness of LLM Code Generation

TL;DR

This work investigates the robustness of large language models in code generation across Java, C++, and JavaScript by extending the ReCode framework with a multilingual EvalPlus-X benchmark. It introduces 29 semantic-preserving perturbations across docstrings, function names, syntax, and formatting, and evaluates six LLMs in a black-box setting, complemented by a docstring-repair mitigation attempt. The study reveals language-specific robustness patterns, showing Java generally more resilient while C++ is more fragile, and demonstrates that model size alone does not guarantee robustness. It also shows that prompt repair offers only modest gains and is not a reliable universal solution, underscoring the need for multilingual robustness benchmarks and targeted training strategies, with future work pointing toward white-box analyses and broader language coverage.

Abstract

Large language models have gained significant traction and popularity in recent times, extending their usage to code-generation tasks. While this field has garnered considerable attention, the exploration of testing and evaluating the robustness of code generation models remains an ongoing endeavor. Previous studies have primarily focused on code generation models specifically for the Python language, overlooking other widely-used programming languages. In this work, we conduct a comprehensive comparative analysis to assess the robustness performance of several prominent code generation models and investigate whether robustness can be improved by repairing perturbed docstrings using an LLM. Furthermore, we investigate how their performance varies across different programming languages. To accomplish this, we introduce perturbations in four key areas of the prompt: DocString, functionname, syntax, and format. We have compiled and released a dedicated dataset for this purpose. This work presents our experimental findings, shedding light on the performance of code generation models in various scenarios.
Paper Structure (31 sections, 4 equations, 10 figures, 16 tables)

This paper contains 31 sections, 4 equations, 10 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Background
  4. Related Work
  5. Methodology
  6. Study Overview
  7. Dataset Preparation
  8. Semantic Preserved Perturbation
  9. Perturbation on DocString
  10. Perturbation on Function Name
  11. Perturbation on Syntax
  12. Perturbation on Format
  13. Evaluation
  14. Robust Pass ())
  15. Robust Drop ()
  16. ...and 16 more sections

Figures (10)

  • Figure 1: A modification in the function name causes incorrect code generation by CodeGen-6B-Multi model
  • Figure 2: A generated code by CodeGen-6B-Multi model which passes the test cases of HumanEval-X but fails on the test cases from EvalPlus.
  • Figure 3: An example of a generated code by CodeGen-6B-Multi snippet from a sample prompt
  • Figure 4: Overview of the study
  • Figure 5: Syntax Perturbations (JavaScript)
  • ...and 5 more figures