Exploring Multi-Lingual Bias of Large Code Models in Code Generation

TL;DR

This work exposes substantial multi-lingual biases in large code models for code generation across natural languages (English vs Chinese) and programming languages (Python vs C++). It introduces the X-HumanEval-X benchmark and the MEIC instruction-tuning dataset to systematically assess and mitigate these biases. The study shows that prompting via translation can reduce NL bias, but instruction tuning with diverse bilingual data yields the largest gains and the strongest bias reductions, though some bias persists. Collectively, the results underscore the importance of multilingual data diversity and targeted training methods to improve cross-language code-generation performance and accessibility. The findings inform researchers and developers on effective strategies to reduce multi-lingual bias and enhance practical applicability of LCMs in diverse linguistic and programming contexts.

Abstract

Code generation aims to synthesize code and fulfill functional requirements based on natural language (NL) specifications, which can greatly improve development efficiency. In the era of large language models (LLMs), large code models (LCMs) have been recently proposed to generate source code. LCMs can generate highly feasible solutions for programming problems described in natural language. Despite the effectiveness, we observe a noticeable multilingual bias in the generation performance of LCMs. Specifically, LCMs demonstrate proficiency in generating solutions when provided with instructions in English, yet may falter when faced with semantically equivalent instructions in other NLs such as Chinese. Moreover, the ability of LCMs to generate code exhibits variety across different programming languages (PLs), such as Python and C++. The observed phenomenon indicates the presence of multi-lingual bias within the generative capabilities of LCMs, which has remained unexplored. In this paper, we aim to investigate the multi-lingual bias that exists in current LCMs. First, we initiate our investigation by constructing the first multi-lingual evaluation benchmark X-HumanEval-X, enabling us to systematically evaluate the extent of multi-lingual bias that exists in current LCMs. In our large-scale experiments on nine popular LCMs, we observe a pronounced multi-lingual bias of LCMs in code generation, including multi-NL and multi-PL bias. Specifically, when using Chinese instructions, the code generation capabilities of LCMs decrease by at least 13% in terms of the Pass@1 metric. Furthermore, LCMs perform variously across different programming languages, e.g., the performance gap between Python and C++ reaches as high as 20.9%. ...

Figures (1)

• Figure 1: Motivation example of the bias that exists among multiple natural languages ((a) and (b)) and programming languages ((c) and (d)). The experimented LCM is DeepSeek-Coder 33B.