RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories

TL;DR

RealSec-bench tackles the problem of evaluating secure code generation in real software contexts by grounding assessment in 105 vulnerability instances mined from real high-risk Java projects. It employs a two-phase construction pipeline (SAST-based vulnerability extraction and human-in-the-loop refinement) and evaluates five LLMs with Pass@$k$, Secure@$k$, and SecurePass@$k$ metrics under retrieval augmentation and security-guided prompts. Key findings show current models struggle to produce code that is simultaneously functionally correct and secure, especially for cryptography and multi-hop dataflows; RAG improves functional accuracy but has limited security benefits, and prompt-based security guidelines yield inconsistent improvements. The work highlights a critical gap in secure code generation and suggests directions toward more robust ground truth, evaluation practices, and cross-language scalability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area. Existing benchmarks often fall short by relying on synthetic vulnerabilities or evaluating functional correctness in isolation, failing to capture the complex interplay between functionality and security found in real-world software. To address this gap, we introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories. Our methodology employs a multi-stage pipeline that combines systematic SAST scanning with CodeQL, LLM-based false positive elimination, and rigorous human expert validation. The resulting benchmark contains 105 instances grounded in real-word repository contexts, spanning 19 Common Weakness Enumeration (CWE) types and exhibiting a wide diversity of data flow complexities, including vulnerabilities with up to 34-hop inter-procedural dependencies. Using RealSec-bench, we conduct an extensive empirical study on 5 popular LLMs. We introduce a novel composite metric, SecurePass@K, to assess both functional correctness and security simultaneously. We find that while Retrieval-Augmented Generation (RAG) techniques can improve functional correctness, they provide negligible benefits to security. Furthermore, explicitly prompting models with general security guidelines often leads to compilation failures, harming functional correctness without reliably preventing vulnerabilities. Our work highlights the gap between functional and secure code generation in current LLMs.

Figures (7)

• Figure 1: Benchmark Construction Overview.
• Figure 2: An Example Task Instance
• Figure 3: Two-stage Security Metric Pipeline.
• Figure 4: Performance analysis in inter-procedural dependency complexity.
• Figure 5: Performance comparison of different retrieval methods (BM25, Dense, Dataflow) across various large language models. The "origin" method represents the baseline performance without any retrieval.