Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories

Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, Yizheng Chen

TL;DR

SecRepoBench tackles secure code completion in real-world C/C++ repositories, integrating repository-wide context with both correctness and security evaluation. It constructs 318 tasks from the ARVO vulnerability dataset, masking secure regions and validating outputs with developer tests and OSS-Fuzz PoCs. The study shows code agents substantially outperform standalone LLMs, but both remain far from perfect, with major gains in correctness and limited gains in security, highlighting missed vulnerabilities and compilation issues. The benchmark thus provides a realistic, scalable framework to guide future improvements in secure, correct code generation for real software projects.

Abstract

This paper introduces SecRepoBench, a benchmark to evaluate code agents on secure code completion in real-world repositories. SecRepoBench has 318 code completion tasks in 27 C/C++ repositories, covering 15 CWEs. We evaluate 28 standalone LLMs and 13 code agents across 3 state-of-the-art agent frameworks using our benchmark. We find that state-of-the-art LLMs struggle with generating correct and secure code completions. However, code agents significantly outperform standalone LLMs. We show that SecRepoBench is more difficult than the prior state-of-the-art benchmark. Finally, our comprehensive analysis provides insights into potential directions for enhancing the ability of code agents to write correct and secure code in real-world repositories.

SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories

TL;DR

SecRepoBench tackles secure code completion in real-world C/C++ repositories, integrating repository-wide context with both correctness and security evaluation. It constructs 318 tasks from the ARVO vulnerability dataset, masking secure regions and validating outputs with developer tests and OSS-Fuzz PoCs. The study shows code agents substantially outperform standalone LLMs, but both remain far from perfect, with major gains in correctness and limited gains in security, highlighting missed vulnerabilities and compilation issues. The benchmark thus provides a realistic, scalable framework to guide future improvements in secure, correct code generation for real software projects.

Abstract

This paper introduces SecRepoBench, a benchmark to evaluate code agents on secure code completion in real-world repositories. SecRepoBench has 318 code completion tasks in 27 C/C++ repositories, covering 15 CWEs. We evaluate 28 standalone LLMs and 13 code agents across 3 state-of-the-art agent frameworks using our benchmark. We find that state-of-the-art LLMs struggle with generating correct and secure code completions. However, code agents significantly outperform standalone LLMs. We show that SecRepoBench is more difficult than the prior state-of-the-art benchmark. Finally, our comprehensive analysis provides insights into potential directions for enhancing the ability of code agents to write correct and secure code in real-world repositories.
Paper Structure (12 sections, 2 figures, 3 tables)

This paper contains 12 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. SecRepoBench
  3. Overview
  4. Benchmark Construction
  5. Comparison against Prior Benchmarks
  6. Experiments
  7. Evaluation Setups
  8. Results
  9. Failure Modes Analysis
  10. Related Work
  11. Discussions
  12. Conclusion

Figures (2)

  • Figure 1: Overview of the SecRepoBench framework.
  • Figure 2: Comparison of secure-pass@$1$ performance on BaxBench versus SecRepoBench. While standalone models achieve relatively high secure-pass@$1$ on BaxBench for generating self-contained programs, their performance significantly declines on the more challenging repository-level benchmark SecRepoBench. Notably, model rankings vary across the two benchmarks, demonstrating that performance on BaxBench does not generalize well to repository-level code generation tasks in SecRepoBench.