YABLoCo: Yet Another Benchmark for Long Context Code Generation

TL;DR

The paper addresses the challenge of evaluating long-context code generation for C/C++ in large repositories by introducing YABLoCo, a benchmark comprising 215 function examples from four sizeable codebases with associated metadata and call graphs. It provides a scalable evaluation pipeline and a visualization tool to measure performance under no-context and oracle-context settings, using baselines such as CodeLlama, DeepSeekCoder, and GPT-4. Experimental results reveal moderate baseline performance without repository context and clear gains when repository context is supplied, illustrating the potential of context-aware prompting for large-scale code generation. The study discusses data leakage, docstring quality, and the need for retrieval-augmented strategies, positioning YABLoCo as a practical platform for advancing long-context code generation research in real-world software engineering. Overall, YABLoCo fills a critical gap by enabling rigorous assessment of LLMs’ ability to generate runnable code with extended cross-file and cross-repository context in C/C++.

Abstract

Large Language Models demonstrate the ability to solve various programming tasks, including code generation. Typically, the performance of LLMs is measured on benchmarks with small or medium-sized context windows of thousands of lines of code. At the same time, in real-world software projects, repositories can span up to millions of LoC. This paper closes this gap by contributing to the long context code generation benchmark (YABLoCo). The benchmark featured a test set of 215 functions selected from four large repositories with thousands of functions. The dataset contained metadata of functions, contexts of the functions with different levels of dependencies, docstrings, functions bodies, and call graphs for each repository. This paper presents three key aspects of the contribution. First, the benchmark aims at function body generation in large repositories in C and C++, two languages not covered by previous benchmarks. Second, the benchmark contains large repositories from 200K to 2,000K LoC. Third, we contribute a scalable evaluation pipeline for efficient computing of the target metrics and a tool for visual analysis of generated code. Overall, these three aspects allow for evaluating code generation in large repositories in C and C++.

Figures (3)

• Figure 1: The counts of functions filtered at every stage
• Figure 2: Overview of the pipeline for testing generated functions and pass@k metric evaluation
• Figure 3: Visualization tool shows an original code snippet on the left and a series of ten generated snippets on the right.