Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation

Musfiqur Rahman, SayedHassan Khatoonabadi, Emad Shihab

TL;DR

This work reveals a substantial gap between synthetic benchmark performance and real-world class-level code generation by LLMs. By constructing RealClassEval with seen and unseen partitions and evaluating seven diverse LLMs across docstring and RAG configurations, the study shows real-world class-level correctness plummets from ~84–89% on synthetic tasks to ~25–34% on real-world tasks, with minimal difference between seen and unseen data. Docstrings yield only modest, often non-significant improvements; however, retrieval augmentation provides meaningful gains—4–7%—particularly when documentation is incomplete, supporting an information gap hypothesis. Error analysis identifies AttributeError, TypeError, and AssertionError as the dominant failure modes, with distinct patterns between synthetic and real-world settings and RAG-induced error substitutions that reduce some errors but introduce dependency-related ones. The findings suggest targeted improvements in object-oriented semantic understanding and context-aware retrieval, and they argue for more realistic, multi-faceted benchmarks to guide production-ready code-generation tools.

Abstract

Large language models (LLMs) have demonstrated strong performance on function-level code generation benchmarks, yet real-world software development increasingly demands class-level implementations that integrate multiple methods, attributes, and dependencies within authentic project contexts. This gap between benchmark performance and practical utility raises critical questions about LLMs' readiness for production code assistance, particularly regarding their ability to generalize across familiar and novel codebases. We introduce a benchmark derived from real-world open-source repositories, comprising classes divided into seen and unseen partitions to evaluate generalization under practical conditions. We systematically examine how input specification completeness and retrieval-augmented generation affect class-level correctness across multiple state-of-the-art LLMs. Our evaluation reveals a substantial performance gap: while LLMs achieve 84 to 89% correctness on synthetic benchmarks, they attain only 25 to 34% on real-world class tasks, with minimal distinction between familiar and novel codebases. Comprehensive documentation provides marginal improvements (1 to 3%), whereas retrieval augmentation yields greater gains (4 to 7%) by supplying concrete implementation patterns. Error analysis identifies AttributeError, TypeError, and AssertionError as dominant failure modes, with distinct patterns between synthetic and real-world scenarios. These findings provide actionable insights for enhancing context modelling, documentation strategies, and retrieval integration in production code assistance tools.

Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation

TL;DR

This work reveals a substantial gap between synthetic benchmark performance and real-world class-level code generation by LLMs. By constructing RealClassEval with seen and unseen partitions and evaluating seven diverse LLMs across docstring and RAG configurations, the study shows real-world class-level correctness plummets from ~84–89% on synthetic tasks to ~25–34% on real-world tasks, with minimal difference between seen and unseen data. Docstrings yield only modest, often non-significant improvements; however, retrieval augmentation provides meaningful gains—4–7%—particularly when documentation is incomplete, supporting an information gap hypothesis. Error analysis identifies AttributeError, TypeError, and AssertionError as the dominant failure modes, with distinct patterns between synthetic and real-world settings and RAG-induced error substitutions that reduce some errors but introduce dependency-related ones. The findings suggest targeted improvements in object-oriented semantic understanding and context-aware retrieval, and they argue for more realistic, multi-faceted benchmarks to guide production-ready code-generation tools.

Abstract

Large language models (LLMs) have demonstrated strong performance on function-level code generation benchmarks, yet real-world software development increasingly demands class-level implementations that integrate multiple methods, attributes, and dependencies within authentic project contexts. This gap between benchmark performance and practical utility raises critical questions about LLMs' readiness for production code assistance, particularly regarding their ability to generalize across familiar and novel codebases. We introduce a benchmark derived from real-world open-source repositories, comprising classes divided into seen and unseen partitions to evaluate generalization under practical conditions. We systematically examine how input specification completeness and retrieval-augmented generation affect class-level correctness across multiple state-of-the-art LLMs. Our evaluation reveals a substantial performance gap: while LLMs achieve 84 to 89% correctness on synthetic benchmarks, they attain only 25 to 34% on real-world class tasks, with minimal distinction between familiar and novel codebases. Comprehensive documentation provides marginal improvements (1 to 3%), whereas retrieval augmentation yields greater gains (4 to 7%) by supplying concrete implementation patterns. Error analysis identifies AttributeError, TypeError, and AssertionError as dominant failure modes, with distinct patterns between synthetic and real-world scenarios. These findings provide actionable insights for enhancing context modelling, documentation strategies, and retrieval integration in production code assistance tools.

Paper Structure

This paper contains 97 sections, 4 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Large Language Models for Code Generation
  4. Benchmarks for Code Generation
  5. Retrieval-Augmented Generation for Code
  6. Docstrings and Prompt Engineering
  7. Methodology
  8. Dataset Curation
  9. Data Collection:
  10. Data Filtering:
  11. Static Analysis and Class Extraction:
  12. Post-processing and Sampling:
  13. Code Generation
  14. Selected LLMs
  15. Prompt Design
  16. ...and 82 more sections

Figures (5)

  • Figure 1: Workflow for RealClassEval dataset curation.
  • Figure 2: Performance of different LLMs on ClassEval, Pre-cutoff, and Post-cutoff data.
  • Figure 3: Violin plot of per-class pass rate distributions across docstring conditions (full, partial, no) for Pre-Cutoff and Post-Cutoff datasets, aggregated across all models.
  • Figure 4: Distribution of Error Types Across Datasets.
  • Figure 5: Error Type Changes with RAG Implementation.