MdEval: Massively Multilingual Code Debugging

TL;DR

MdEval introduces a massively multilingual code debugging benchmark spanning 20 programming languages and three tasks (APR, BL, BI) to evaluate LLMs beyond Python-focused benchmarks. It provides MdEval-Instruct, an instruction corpus generated by xDebugGen, and a multilingual baseline xDebugCoder trained with instruction-tuning data. Experiments across 40 models reveal a substantial performance gap between open-source and closed-source LLMs, underscoring the need for broader multilingual debugging capabilities and improved instruction-following. The work also analyzes error-type sensitivity, effect of bug location, and alternative APR settings, offering a foundation for future multilingual debugging research and benchmark expansion.

Abstract

Code large language models (LLMs) have made significant progress in code debugging by directly generating the correct code based on the buggy code snippet. Programming benchmarks, typically consisting of buggy code snippet and their associated test cases, are used to assess the debugging capabilities of LLMs. However, many existing benchmarks primarily focus on Python and are often limited in terms of language diversity (e.g., DebugBench and DebugEval). To advance the field of multilingual debugging with LLMs, we propose the first massively multilingual debugging benchmark, which includes 3.6K test samples of 18 programming languages and covers the automated program repair (APR) task, the code review (CR) task, and the bug identification (BI) task. Further, we introduce the debugging instruction corpora MDEVAL-INSTRUCT by injecting bugs into the correct multilingual queries and solutions (xDebugGen). Further, a multilingual debugger xDebugCoder trained on MDEVAL-INSTRUCT as a strong baseline specifically to handle the bugs of a wide range of programming languages (e.g. "Missing Mut" in language Rust and "Misused Macro Definition" in language C). Our extensive experiments on MDEVAL reveal a notable performance gap between open-source models and closed-source LLMs (e.g., GPT and Claude series), highlighting huge room for improvement in multilingual code debugging scenarios.

Figures (9)

• Figure 1: Massively multilingual evaluation task comprised of three tasks, including code generation, code completion, and code explanation.
• Figure 2: Error types in MdEval. Part (a) shows generic error types, and Part (b) lists language-specific error types.
• Figure 3: Error types distribution in 20 programming Languages from the MdEval.
• Figure 4: Overview of the MdEval construction process. We collected and filtered code snippets from GitHub. Before annotation, we summarized error types. Annotators then labeled the code based on these types. To ensure quality, they used GPT-4o to evaluate the annotations on four criteria: difficulty, ambiguity, error type, and solvable. Finally, they exchanged data with each other to minimize bias and errors.
• Figure 5: Examples of multilingual automated program repair (APR), bug localization (BL), and code identification (CI).