Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing and Improving the Representativeness of Code Generation Benchmarks Using Knowledge Units (KUs) of Programming Languages -- An Empirical Study

Md Ahasanuzzaman, Bram Adams, Emad Fallahzadeh, Gustavo A. Oliva, Ahmed E. Hassan

TL;DR

This study exposes substantial gaps in how Python code generation benchmarks reflect real world programming concepts by framing concepts as Knowledge Units (KUs). Using an LLM based detector, it compares HumanEval and MBPP against 30 real world Python projects and finds that benchmarks cover only about half of the identified 20 KUs and exhibit skewed distributions. To address this, the authors synthesize 440 KU based tasks across 11 underrepresented KUs and create augmented benchmarks, achieving over a 60% improvement in distributional alignment with real world KU usage as measured by Jensen–Shannon Distance. Evaluations across seven diverse LLMs show consistent performance drops on augmented benchmarks, highlighting that current benchmarks overestimate code generation capabilities due to limited KU coverage. The work provides actionable guidance for constructing more realistic, KU aware benchmarks and for diagnosing model strengths and weaknesses with respect to concrete programming language concepts.

Abstract

Large Language Models (LLMs) such as GPT-4, Claude and LLaMA have shown impressive performance in code generation, typically evaluated using benchmarks (e.g., HumanEval). However, effective code generation requires models to understand and apply a wide range of language concepts. If the concepts exercised in benchmarks are not representative of those used in real-world projects, evaluations may yield incomplete. Despite this concern, the representativeness of code concepts in benchmarks has not been systematically examined. To address this gap, we present the first empirical study that analyzes the representativeness of code generation benchmarks through the lens of Knowledge Units (KUs) - cohesive sets of programming language capabilities provided by language constructs and APIs. We analyze KU coverage in two widely used Python benchmarks, HumanEval and MBPP, and compare them with 30 real-world Python projects. Our results show that each benchmark covers only half of the identified 20 KUs, whereas projects exercise all KUs with relatively balanced distributions. In contrast, benchmark tasks exhibit highly skewed KU distributions. To mitigate this misalignment, we propose a prompt-based LLM framework that synthesizes KU-based tasks to rebalance benchmark KU distributions and better align them with real-world usage. Using this framework, we generate 440 new tasks and augment existing benchmarks. The augmented benchmarks substantially improve KU coverage and achieve over a 60% improvement in distributional alignment. Evaluations of state-of-the-art LLMs on these augmented benchmarks reveal consistent and statistically significant performance drops (12.54-44.82%), indicating that existing benchmarks overestimate LLM performance due to their limited KU coverage. Our findings provide actionable guidance for building more realistic evaluations of LLM code-generation capabilities.

Assessing and Improving the Representativeness of Code Generation Benchmarks Using Knowledge Units (KUs) of Programming Languages -- An Empirical Study

TL;DR

This study exposes substantial gaps in how Python code generation benchmarks reflect real world programming concepts by framing concepts as Knowledge Units (KUs). Using an LLM based detector, it compares HumanEval and MBPP against 30 real world Python projects and finds that benchmarks cover only about half of the identified 20 KUs and exhibit skewed distributions. To address this, the authors synthesize 440 KU based tasks across 11 underrepresented KUs and create augmented benchmarks, achieving over a 60% improvement in distributional alignment with real world KU usage as measured by Jensen–Shannon Distance. Evaluations across seven diverse LLMs show consistent performance drops on augmented benchmarks, highlighting that current benchmarks overestimate code generation capabilities due to limited KU coverage. The work provides actionable guidance for constructing more realistic, KU aware benchmarks and for diagnosing model strengths and weaknesses with respect to concrete programming language concepts.

Abstract

Large Language Models (LLMs) such as GPT-4, Claude and LLaMA have shown impressive performance in code generation, typically evaluated using benchmarks (e.g., HumanEval). However, effective code generation requires models to understand and apply a wide range of language concepts. If the concepts exercised in benchmarks are not representative of those used in real-world projects, evaluations may yield incomplete. Despite this concern, the representativeness of code concepts in benchmarks has not been systematically examined. To address this gap, we present the first empirical study that analyzes the representativeness of code generation benchmarks through the lens of Knowledge Units (KUs) - cohesive sets of programming language capabilities provided by language constructs and APIs. We analyze KU coverage in two widely used Python benchmarks, HumanEval and MBPP, and compare them with 30 real-world Python projects. Our results show that each benchmark covers only half of the identified 20 KUs, whereas projects exercise all KUs with relatively balanced distributions. In contrast, benchmark tasks exhibit highly skewed KU distributions. To mitigate this misalignment, we propose a prompt-based LLM framework that synthesizes KU-based tasks to rebalance benchmark KU distributions and better align them with real-world usage. Using this framework, we generate 440 new tasks and augment existing benchmarks. The augmented benchmarks substantially improve KU coverage and achieve over a 60% improvement in distributional alignment. Evaluations of state-of-the-art LLMs on these augmented benchmarks reveal consistent and statistically significant performance drops (12.54-44.82%), indicating that existing benchmarks overestimate LLM performance due to their limited KU coverage. Our findings provide actionable guidance for building more realistic evaluations of LLM code-generation capabilities.
Paper Structure (23 sections, 3 equations, 11 figures, 6 tables)

This paper contains 23 sections, 3 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Knowledge Units (KUs)
  3. Definition
  4. Operationalizing KUs via LLMs
  5. Detection of the incidence of KUs via LLMs
  6. Data Collection and Preparation of the Empirical Study
  7. Empirical study of the language concepts used in code generation
  8. RQ1: How comprehensively do existing LLM benchmark datasets cover language concepts compared to real-world software projects?
  9. RQ1.1: How well do LLM benchmark datasets represent the breadth of KUs found in real-world software projects?
  10. RQ1.2: To what extent are KUs evenly distributed across tasks in existing benchmark datasets compared to real-world software projects?
  11. RQ2: To what extent can synthesized tasks restore the representativeness of language concepts in existing benchmarks?
  12. RQ2.1: To what extent does augmenting existing benchmarks with synthesized KU-based tasks improve their code-concept representativeness with respect to real-world projects?
  13. RQ2.2: How well do LLMs perform in generating code for solving tasks across different KUs?
  14. Implications
  15. Implications for researchers
  16. ...and 8 more sections

Figures (11)

  • Figure 1: Our metamodel for knowledge units (KUs).
  • Figure 3: Percentage of instances of every KU across HumanEval, MBPP and real-world projects.
  • Figure 4: The Lorenz curves of real-world projects and benchmarks that are present in both datasets.
  • Figure 5: Our LLM-based approach to automatically synthesize KU-based tasks
  • Figure 6: Heat map of the pass@1 of the studied models across KUs for the generated KU-based tasks. The color of each cell encodes the pass@1 value, ranging from green (lower values) to yellow (higher values).
  • ...and 6 more figures