RepoMasterEval: Evaluating Code Completion via Real-World Repositories

TL;DR

RepoMasterEval addresses the misalignment between existing code completion benchmarks and real-world development by constructing a repository-centered Fill-In-the-Middle benchmark from GitHub projects. It uses masked code with prefix, suffix, and repository-level context retrieved via BM25, and augments test quality with mutation testing and manual test crafting. The authors evaluate 10 models, showing test augmentation increases mutation scores and reveals robustness gaps; pass rates correlate with real-world deployment acceptance. The benchmark captures diverse tasks across Python and TypeScript and demonstrates its utility for guiding model training and debugging. Overall, RepoMasterEval provides a practical, robust framework for benchmarking code completion in realistic settings.

Abstract

With the growing reliance on automated code completion tools in software development, the need for comprehensive evaluation benchmarks has become critical. Existing benchmarks focus more on code completion in function and class level by providing text descriptions to prompt the model. By contrast, such descriptive prompt is commonly unavailable in real development and code completion can occur in wider range of situations such as in the middle of a function or a code block. These limitations makes existing evaluation benchmarks poorly align with the practical scenarios of code completion tools. In this paper, we propose RepoMasterEval, a novel benchmark for evaluating code completion models constructed from real-world repositories. Each benchmark datum is generated by masking a code snippet (ground truth) from one source code file with existing test suites. To improve test accuracy of model generated code, we employ mutation testing to measure the effectiveness of the test cases and we manually crafted new test cases for those test suites with low mutation score. Our empirical evaluation on 10 state-of-the-art models shows that test argumentation is critical in improving the accuracy of the benchmark and RepoMasterEval is able to report variance in model performance in real-world scenarios. The deployment of RepoMasterEval also revealed that the benchmark is useful to give accurate feedback during model training and the score is in high correlation with the model's performance in practice.

Figures (1)

• Figure 1: A figure with two subfigures