AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

TL;DR

AutoCodeBench delivers a fully automated, annotation-free framework to build large-scale multilingual code benchmarks by coupling LLM-driven solution and test-function generation with a multilingual sandbox. The AutoCodeGen workflow synthesizes self-contained solutions, verifiable test inputs/outputs, and diverse programming problems across 20 languages, producing 3,920 problems (plus Lite and Complete variants). Evaluations across 30+ models reveal persistent difficulty in multilingual, multi-logic tasks, with notable gains from reasoning modes and sandbox feedback, and reveal biases in automated generation that the authors discuss mitigation for. The study provides an open multilingual sandbox and a scalable benchmark resource that highlights current model gaps and offers actionable directions for improving multilingual code generation capabilities and evaluation practices.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, with code generation emerging as a key area of focus. While numerous benchmarks have been proposed to evaluate their code generation abilities, these benchmarks face several critical limitations. First, they often rely on manual annotations, which are time-consuming and difficult to scale across different programming languages and problem complexities. Second, most existing benchmarks focus primarily on Python, while the few multilingual benchmarks suffer from limited difficulty and uneven language distribution. To address these challenges, we propose AutoCodeGen, an automated method for generating high-difficulty multilingual code generation datasets without manual annotations. AutoCodeGen ensures the correctness and completeness of test cases by generating test inputs with LLMs and obtaining test outputs through a multilingual sandbox, while achieving high data quality through reverse-order problem generation and multiple filtering steps. Using this novel method, we introduce AutoCodeBench, a large-scale code generation benchmark comprising 3,920 problems evenly distributed across 20 programming languages. It is specifically designed to evaluate LLMs on challenging, diverse, and practical multilingual tasks. We evaluate over 30 leading open-source and proprietary LLMs on AutoCodeBench and its simplified version AutoCodeBench-Lite. The results show that even the most advanced LLMs struggle with the complexity, diversity, and multilingual nature of these tasks. Besides, we introduce AutoCodeBench-Complete, specifically designed for base models to assess their few-shot code generation capabilities. We hope the AutoCodeBench series will serve as a valuable resource and inspire the community to focus on more challenging and practical multilingual code generation scenarios.

Figures (15)

• Figure 1: Tag and Language Distribution across our AutoCodeBench.
• Figure 2: The overview of AutoCodeGen. It first generates code solution and the corresponding public/private test input functions based on multilingual code snippets (①). They are concatenated and executed in a sandbox to obtain test outputs, which are then combined by the LLM into complete test functions (②,③,④). Based on the code solution and test function, the LLM is prompted to generate accurate programming problems (⑤). Finally, a three-stage data filtering is applied: multiple sampling to remove too easy problems (⑥), LLM-as-Critic to discard low-quality ones (⑦), and diversity-based tagging to ensure distributional variety (⑧).
• Figure 3: The performance comparison of different models across two language sets.
• Figure 4: Performance drop of models on multi-logic problems (1,622) compared to full dataset.
• Figure 5: Scaling laws for different models.