Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs

TL;DR

Defects4C addresses the lack of high-quality C/C++ repair benchmarks by introducing a large-scale, executable dataset derived from real-world repositories, including 9M bug-related commits and 350 validated faults (248 bugs +102 vulnerabilities) with test cases. The authors deploy a rigorous, multi-stage construction pipeline (raw commit collection, unit-test matching, and human validation) and provide tooling for scalable patch extraction and verification. An extensive empirical study evaluates 24 pre-trained LLMs and fine-tuned models on C/C++ repair, revealing a substantial performance gap relative to Java benchmarks and highlighting the need for C/C++-specific repair techniques and advanced fine-tuning strategies. Overall, Defects4C serves as both a rigorous evaluation benchmark and a data resource for training and improving LLM-based program repair for real-world C/C++ software.

Abstract

Automated Program Repair (APR) plays a critical role in enhancing the quality and reliability of software systems. While substantial progress has been made in Java-based APR, largely facilitated by benchmarks like Defects4J, there remains a significant gap in research on C/C++ program repair, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities. This gap is primarily due to the lack of high-quality, open-source benchmarks tailored for C/C++. To address this issue, we introduce Defects4C, a comprehensive and executable benchmark specifically designed for C/C++ program repair. Our dataset is constructed from real-world C/C++ repositories and includes a large collection of bug-relevant commits (9M in total), 248 high-quality buggy functions, and 102 vulnerable functions, all paired with test cases for reproduction. These resources enable rigorous evaluation of repair techniques and support the retraining of learning-based approaches for enhanced performance. Using Defects4C, we conduct a comprehensive empirical study evaluating the effectiveness of 24 state-of-the-art large language models (LLMs) in repairing C/C++ faults. Our findings offer valuable insights into the strengths and limitations of current LLM-based APR techniques in this domain, highlighting both the need for more robust methods and the critical role of Defects4C in advancing future research

Figures (2)

• Figure 1:
• Figure 2: Prompt design for different types of defects.