EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

TL;DR

EffiBench-X targets the problem of measuring runtime efficiency of LLM-generated code across six languages using competition-style tasks and expert baselines. It introduces a holistic framework with canonical solution construction, robust test-suite generation, sandboxed execution, and high-resolution profiling to quantify efficiency via ET, MP, and MI metrics, with $s_i^T = \operatorname{clip}\left(\frac{T_i^{H}}{T_i^{L}}, 0, 1\right)$ and similar definitions for MP and MI, and ET, MP, MI expressed as aggregate percentages. The study reveals a substantial efficiency gap: the best models reach only around 62% of human-level runtime efficiency on average, with language-specific variance where dynamic languages fare better and larger/specialized models provide notable gains in correctness. These results motivate targeted optimization of multilingual code generation and broader benchmarking coverage, rather than focusing solely on functional accuracy. The dataset and evaluation framework are publicly available at the provided URLs, enabling future research on multilingual, efficiency-oriented code generation.

Abstract

Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To address this gap, we introduce EffiBench-X, the first multi-language benchmark designed to measure the efficiency of LLM-generated code. EffiBench-X supports Python, C++, Java, JavaScript, Ruby, and Golang. It comprises competitive programming tasks with human-expert solutions as efficiency baselines. Evaluating state-of-the-art LLMs on EffiBench-X reveals that while models generate functionally correct code, they consistently underperform human experts in efficiency. Even the most efficient LLM-generated solutions (Qwen3-32B) achieve only around \textbf{62\%} of human efficiency on average, with significant language-specific variations. LLMs show better efficiency in Python, Ruby, and JavaScript than in Java, C++, and Golang. For instance, DeepSeek-R1's Python code is significantly more efficient than its Java code. These results highlight the critical need for research into LLM optimization techniques to improve code efficiency across diverse languages. The dataset and evaluation infrastructure are submitted and available at https://github.com/EffiBench/EffiBench-X.git and https://huggingface.co/datasets/EffiBench/effibench-x.

Figures (5)

• Figure 1: Overview of the construction pipeline. The process begins with collecting efficiency-critical problems from competitive programming platforms, followed by constructing canonical solutions from expert programmers, and then generating test suites with test case generators and solution evaluators.
• Figure 2: This figure shows a Python problem where DeepSeek-R1 generated code with efficiency nearly matching the human expert solution (ratio = 0.985). Both calculate adjacent character ASCII differences using equivalent linear-time algorithms with constant space complexity, demonstrating that well-trained LLMs can identify optimal approaches matching expert solutions.
• Figure 3: This figure shows a C++ problem where Claude-3-7-Sonnet's solution achieves exceptional efficiency by choosing an optimal data structure.
• Figure 4: This figure shows Claude-3.7-Sonnet's inefficient solution using 4D dynamic programming to count strings containing "leet."
• Figure 5: This figure presents the human expert's elegant solution using the Inclusion-Exclusion Principle, which directly calculates the answer by subtracting invalid configurations from the total possible strings.