Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks

TL;DR

This study targets the gap where benchmark performance does not reveal persistent weaknesses of LLMs in code generation. It conducts a task-level empirical analysis across MBPP, HumanEval, BigCodeBench, and LiveCodeBench using six diverse LLMs to identify tasks that repeatedly fail and to examine factors beyond solution code complexity. By introducing static, AST-based code-complexity metrics and correlating them with failure rates, the authors find that complexity explains failures mainly in LiveCodeBench, while other benchmarks show weak or no systematic relationships. A detailed failure inspection of 114 consistently failed tasks uncovers four recurring patterns—wrong problem mapping, flawed or incomplete algorithm design, edge-case mishandling, and formatting mistakes—along with the influence of ambiguous prompts and test design on observed results. The authors publicly release data and analysis scripts, and discuss implications for benchmark design and targeted improvements to LLM training and evaluation.

Abstract

Large Language Models (LLMs) have achieved remarkable success in code generation, and the race to improve their performance has become a central focus of AI research. Benchmarks and leaderboards are increasingly popular, offering quantitative rankings of LLMs. However, they provide limited insight into the tasks that LLMs consistently fail to solve - information that is crucial for understanding current limitations and guiding the development of more capable models. To address this gap, we examined code generation tasks across four popular benchmarks, identifying those that major LLMs are most likely to fail. To understand the causes of these failures, we investigated whether the static complexity of solution code contributes to them, followed by a systematic inspection of 114 tasks that LLMs consistently struggled with. Our analysis revealed four recurring patterns of weaknesses in LLMs, as well as common complications within benchmark tasks that most often lead to failure.

Figures (1)

• Figure 1: Comparison of benchmark code metrics for tasks, grouped by the number of LLMs that failed them. LiveCodeBench shows a positive correlation between code complexity and failure rate, while other benchmarks do not.