M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation

TL;DR

M2G-Eval tackles the problem of benchmarking code generation by introducing a multi-granularity, multilingual framework that evaluates LLMs across Class, Function, Block, and Line levels in 18 languages. It provides a comprehensive pipeline, including M2G-Eval-Instruct for instruction-tuning data and M2G-Eval-Coder models trained with SFT and GRPO, plus a contamination-controlled test set of 1,286 instances. The study reveals a clear difficulty gradient (Line < Block/Function < Class), stronger performance on full-granularity languages, and high cross-language correlations suggesting transferable programming concepts. By enabling fine-grained analysis of code synthesis, M2G-Eval offers a rigorous diagnostic tool for long-form, multi-file code generation and establishes a foundation for future improvements in multilingual code generation benchmarks.

Abstract

The rapid advancement of code large language models (LLMs) has sparked significant research interest in systematically evaluating their code generation capabilities, yet existing benchmarks predominantly assess models at a single structural granularity and focus on limited programming languages, obscuring fine-grained capability variations across different code scopes and multilingual scenarios. We introduce M2G-Eval, a multi-granularity, multilingual framework for evaluating code generation in large language models (LLMs) across four levels: Class, Function, Block, and Line. Spanning 18 programming languages, M2G-Eval includes 17K+ training tasks and 1,286 human-annotated, contamination-controlled test instances. We develop M2G-Eval-Coder models by training Qwen3-8B with supervised fine-tuning and Group Relative Policy Optimization. Evaluating 30 models (28 state-of-the-art LLMs plus our two M2G-Eval-Coder variants) reveals three main findings: (1) an apparent difficulty hierarchy, with Line-level tasks easiest and Class-level most challenging; (2) widening performance gaps between full- and partial-granularity languages as task complexity increases; and (3) strong cross-language correlations, suggesting that models learn transferable programming concepts. M2G-Eval enables fine-grained diagnosis of code generation capabilities and highlights persistent challenges in synthesizing complex, long-form code.

Figures (12)

• Figure 1: M$^{2}$G-Eval provides more challenging, multi-granularity code generation across more programming languages than previous work.
• Figure 2: Four task granularity examples for M$^{2}$G-Eval. Each example uses a simple Python code snippet to illustrate the data composition of Class, Function, Block, and Line-level tasks.
• Figure 3: We construct M$^{2}$G-Eval-Instruct by first curating sources across 18 languages, categorizing the materials, and instantiating four task granularities (class, function, block, line). Each task is wrapped as a structured prompt, after which we perform LLM-based quality filtering to obtain the final M$^{2}$G-Eval-Instruct.
• Figure 4: Task count of $\mathcal{D}_{\text{t}}$ and $\mathcal{D}_{\text{e}}$. The Y-axis is logarithmic; the left side of the dashed line is a partial-granularity group, and the right side is a full-granularity group. The same applies below.
• Figure 5: Task input and output statistics of $\mathcal{D}_{\text{t}}$ and $\mathcal{D}_{\text{e}}$.