A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs

TL;DR

This work addresses the lack of real-world class-level benchmarks for code generation by compiling a large-scale Python dataset derived from 13,174 open-source projects, yielding 842,656 fully processed class skeletons with signatures and docstrings plus 27 static metrics. The authors extract class structures via ASTs, preserve dependencies, and provide accompanying data and metadata for benchmarking. They evaluate LLM-assisted class generation by prompting GPT-4 with skeletons, achieving strong textual and structural similarity to human-written implementations (e.g., BLEU ~0.59, ROUGE@L ~0.80, TSED ~0.73). The results demonstrate the value of rich, context-aware prompts for class-level code synthesis and position the dataset as a foundation for benchmarking, training, and advancing realistic software-engineering applications of LLMs.

Abstract

Recent advancements in large language models (LLMs) have demonstrated promising capabilities in code generation tasks. However, most existing benchmarks focus on isolated functions and fail to capture the complexity of real-world, class-level software structures. To address this gap, we introduce a large-scale, Python class-level dataset curated from $13{,}174$ real-world open-source projects. The dataset contains over 842,000 class skeletons, each including class and method signatures, along with associated docstrings when available. We preserve structural and contextual dependencies critical to realistic software development scenarios and enrich the dataset with static code metrics to support downstream analysis. To evaluate the usefulness of this dataset, we use extracted class skeletons as prompts for GPT-4 to generate full class implementations. Results show that the LLM-generated classes exhibit strong lexical and structural similarity to human-written counterparts, with average ROUGE@L, BLEU, and TSED scores of 0.80, 0.59, and 0.73, respectively. These findings confirm that well-structured prompts derived from real-world class skeletons significantly enhance LLM performance in class-level code generation. This dataset offers a valuable resource for benchmarking, training, and improving LLMs in realistic software engineering contexts.