Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection

Richard A. Dubniczky, Krisztofer Zoltán Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C. Cordeiro, Norbert Tihanyi

TL;DR

CASTLE introduces a standardized, compilable C benchmark of 250 micro-bPrograms across 25 CWEs to evaluate vulnerability detection across static analyzers, formal verification tools, and LLMs. A novel CASTLE Score integrates true/false positives with CWE severity-derived bonuses, and CASTLE Combination Score assesses multi-tool ensembles, revealing that LLMs excel on small code while failing as code size grows, whereas formal verifiers offer low false positives but narrow vulnerability coverage. The study demonstrates that tool combinations can outperform single tools by offsetting weaknesses, and that LLMs hold promise for real-time security guidance in code completion contexts. The dataset and evaluation framework provide a reproducible, cross-method basis for assessing current and future vulnerability detection approaches.

Abstract

Identifying vulnerabilities in source code is crucial, especially in critical software components. Existing methods such as static analysis, dynamic analysis, formal verification, and recently Large Language Models are widely used to detect security flaws. This paper introduces CASTLE (CWE Automated Security Testing and Low-Level Evaluation), a benchmarking framework for evaluating the vulnerability detection capabilities of different methods. We assess 13 static analysis tools, 10 LLMs, and 2 formal verification tools using a hand-crafted dataset of 250 micro-benchmark programs covering 25 common CWEs. We propose the CASTLE Score, a novel evaluation metric to ensure fair comparison. Our results reveal key differences: ESBMC (a formal verification tool) minimizes false positives but struggles with vulnerabilities beyond model checking, such as weak cryptography or SQL injection. Static analyzers suffer from high false positives, increasing manual validation efforts for developers. LLMs perform exceptionally well in the CASTLE dataset when identifying vulnerabilities in small code snippets. However, their accuracy declines, and hallucinations increase as the code size grows. These results suggest that LLMs could play a pivotal role in future security solutions, particularly within code completion frameworks, where they can provide real-time guidance to prevent vulnerabilities. The dataset is accessible at https://github.com/CASTLE-Benchmark.

CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection

TL;DR

CASTLE introduces a standardized, compilable C benchmark of 250 micro-bPrograms across 25 CWEs to evaluate vulnerability detection across static analyzers, formal verification tools, and LLMs. A novel CASTLE Score integrates true/false positives with CWE severity-derived bonuses, and CASTLE Combination Score assesses multi-tool ensembles, revealing that LLMs excel on small code while failing as code size grows, whereas formal verifiers offer low false positives but narrow vulnerability coverage. The study demonstrates that tool combinations can outperform single tools by offsetting weaknesses, and that LLMs hold promise for real-time security guidance in code completion contexts. The dataset and evaluation framework provide a reproducible, cross-method basis for assessing current and future vulnerability detection approaches.

Abstract

Identifying vulnerabilities in source code is crucial, especially in critical software components. Existing methods such as static analysis, dynamic analysis, formal verification, and recently Large Language Models are widely used to detect security flaws. This paper introduces CASTLE (CWE Automated Security Testing and Low-Level Evaluation), a benchmarking framework for evaluating the vulnerability detection capabilities of different methods. We assess 13 static analysis tools, 10 LLMs, and 2 formal verification tools using a hand-crafted dataset of 250 micro-benchmark programs covering 25 common CWEs. We propose the CASTLE Score, a novel evaluation metric to ensure fair comparison. Our results reveal key differences: ESBMC (a formal verification tool) minimizes false positives but struggles with vulnerabilities beyond model checking, such as weak cryptography or SQL injection. Static analyzers suffer from high false positives, increasing manual validation efforts for developers. LLMs perform exceptionally well in the CASTLE dataset when identifying vulnerabilities in small code snippets. However, their accuracy declines, and hallucinations increase as the code size grows. These results suggest that LLMs could play a pivotal role in future security solutions, particularly within code completion frameworks, where they can provide real-time guidance to prevent vulnerabilities. The dataset is accessible at https://github.com/CASTLE-Benchmark.

Paper Structure

This paper contains 16 sections, 2 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Datasets and Benchmarks
  4. Traditional Vulnerability Scanning Overview
  5. LLM-Based Vulnerability Detection
  6. Methodology
  7. Dataset
  8. Test Format and Wrappers
  9. CASTLE Score
  10. CASTLE Combination Score
  11. Discussion
  12. Tool evaluation on CASTLE Benchmark
  13. LLM Evaluation on CASTLE Benchmark
  14. Limitations
  15. Conclusion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: The CASTLE Benchmark Framework.
  • Figure 2: An example of a micro-benchmark illustrating a buffer overflow
  • Figure 3: Prompt Template for LLMs for JSON-formatted vulnerability reports
  • Figure 4: CASTLE Scores for tools tested on $250$ C programs, including the top five tool combinations. Tools reporting no issues score $200$ points. The theoretical maximum of a perfect score is $1250$ points.
  • Figure 5: True positive vs False positive rates of tools