Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization

Qiwei Peng, Yekun Chai, Xuhong Li

TL;DR

HumanEval-XL tackles the lack of multilingual NL-to-code evaluation by pairing 23 NLs with 12 PLs and producing 22,080 parallel prompts. The authors implement a four-stage back-translation–based data construction pipeline using GPT-4, with BERTScore filtering and manual quality checks, to create a robust, shared benchmark. Through experiments with CodeGen2, CodeT5+, GPT-3.5, and GPT-4, they show that model size and task-specific pre-training influence code generation across languages, with GPT-4 often leading but notable gaps in non-Python languages like Scala and Go. The work highlights persistent cross-lingual NL semantic gaps and establishes a public resource to drive future improvements in multilingual code generation.

Abstract

Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/humaneval-xl}.

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization

TL;DR

HumanEval-XL tackles the lack of multilingual NL-to-code evaluation by pairing 23 NLs with 12 PLs and producing 22,080 parallel prompts. The authors implement a four-stage back-translation–based data construction pipeline using GPT-4, with BERTScore filtering and manual quality checks, to create a robust, shared benchmark. Through experiments with CodeGen2, CodeT5+, GPT-3.5, and GPT-4, they show that model size and task-specific pre-training influence code generation across languages, with GPT-4 often leading but notable gaps in non-Python languages like Scala and Go. The work highlights persistent cross-lingual NL semantic gaps and establishes a public resource to drive future improvements in multilingual code generation.

Abstract

Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/humaneval-xl}.
Paper Structure (24 sections, 2 figures, 16 tables)

This paper contains 24 sections, 2 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Code Generation Benchmarks
  4. Multilingual Code LLMs
  5. HumanEval-XL
  6. Design Principles
  7. (1) Task Complexity.
  8. (2) Language Diversity.
  9. (3) Accessibility.
  10. Dataset Construction
  11. (1) Extraction and Wrapping of NL Prompt:
  12. (2) Translation and Back-Translation:
  13. (3) Quality Check with BERTScore:
  14. (4) Quality Control:
  15. PLs and NLs
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Illustration of data construction in four steps: (a) Text Extraction (Stage 1): We extract NL texts from the prompt in stage 1. (b) Translation and Back-Translation (Stage 2): The extracted texts are translated into 23 different languages using GPT-4. These translations are then back-translated to English for subsequent automatic quality checks. (c) Quality Assessment with BERTScore (Stage 3): Stage 3 assesses translation quality by computing the BERTScore similarity score between the original English text and its back-translated text. Translations with a low similarity score (threshold $< 0.95$) are rejected and subjected to re-translation. (d) Quality Control (Stage 4): Heuristic checks and manual evaluations are performed on the quality of the translated texts.
  • Figure 2: Performance of LLMs, including CodeT5+ (2B), CodeGen2 (3.7B, 16B), GPT-3.5, and GPT-4, on our proposed benchmark. We report pass@1 metric for evaluation. We order languages in their resource availability as summarized in CC100 XL corpus lin2022few.