Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities

Ziyang Li, Saikat Dutta, Mayur Naik

TL;DR

IRIS addresses security vulnerability detection by combining LLM-driven taint-specification inference with static taint analysis to enable whole-repository reasoning. It introduces CWE-Bench-Java, a curated dataset of 120 real-world Java vulnerabilities, and demonstrates that IRIS with GPT-4 detects more vulnerabilities and lowers false positives than CodeQL, while also uncovering previously unknown issues. The approach leverages LLMs to infer dynamic sources/sinks, CodeQL for precise taint propagation, and contextual prompt-based triage to prune alarms. The work advances practical software security analysis by reducing manual specification effort and enabling scalable, project-wide vulnerability detection.

Abstract

Software is prone to security vulnerabilities. Program analysis tools to detect them have limited effectiveness in practice due to their reliance on human labeled specifications. Large language models (or LLMs) have shown impressive code generation capabilities but they cannot do complex reasoning over code to detect such vulnerabilities especially since this task requires whole-repository analysis. We propose IRIS, a neuro-symbolic approach that systematically combines LLMs with static analysis to perform whole-repository reasoning for security vulnerability detection. Specifically, IRIS leverages LLMs to infer taint specifications and perform contextual analysis, alleviating needs for human specifications and inspection. For evaluation, we curate a new dataset, CWE-Bench-Java, comprising 120 manually validated security vulnerabilities in real-world Java projects. A state-of-the-art static analysis tool CodeQL detects only 27 of these vulnerabilities whereas IRIS with GPT-4 detects 55 (+28) and improves upon CodeQL's average false discovery rate by 5% points. Furthermore, IRIS identifies 4 previously unknown vulnerabilities which cannot be found by existing tools. IRIS is available publicly at https://github.com/iris-sast/iris.

IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities

TL;DR

IRIS addresses security vulnerability detection by combining LLM-driven taint-specification inference with static taint analysis to enable whole-repository reasoning. It introduces CWE-Bench-Java, a curated dataset of 120 real-world Java vulnerabilities, and demonstrates that IRIS with GPT-4 detects more vulnerabilities and lowers false positives than CodeQL, while also uncovering previously unknown issues. The approach leverages LLMs to infer dynamic sources/sinks, CodeQL for precise taint propagation, and contextual prompt-based triage to prune alarms. The work advances practical software security analysis by reducing manual specification effort and enabling scalable, project-wide vulnerability detection.

Abstract

Software is prone to security vulnerabilities. Program analysis tools to detect them have limited effectiveness in practice due to their reliance on human labeled specifications. Large language models (or LLMs) have shown impressive code generation capabilities but they cannot do complex reasoning over code to detect such vulnerabilities especially since this task requires whole-repository analysis. We propose IRIS, a neuro-symbolic approach that systematically combines LLMs with static analysis to perform whole-repository reasoning for security vulnerability detection. Specifically, IRIS leverages LLMs to infer taint specifications and perform contextual analysis, alleviating needs for human specifications and inspection. For evaluation, we curate a new dataset, CWE-Bench-Java, comprising 120 manually validated security vulnerabilities in real-world Java projects. A state-of-the-art static analysis tool CodeQL detects only 27 of these vulnerabilities whereas IRIS with GPT-4 detects 55 (+28) and improves upon CodeQL's average false discovery rate by 5% points. Furthermore, IRIS identifies 4 previously unknown vulnerabilities which cannot be found by existing tools. IRIS is available publicly at https://github.com/iris-sast/iris.
Paper Structure (36 sections, 3 equations, 11 figures, 9 tables)

This paper contains 36 sections, 3 equations, 11 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. IRIS Framework
  4. Problem Statement
  5. Candidate Source/Sink Extraction
  6. Inferring Taint Specifications using LLMs
  7. Vulnerability Detection
  8. Triaging of Alerts via Contextual Analysis
  9. Evaluation Metrics
  10. CWE-Bench-Java: A Dataset of Security Vulnerabilities in Java
  11. Evaluation
  12. Experimental Setup
  13. RQ1: Effectiveness of IRIS on Detecting Existing Vulnerabilities
  14. RQ2: Previously Unknown Vulnerabilities by IRIS
  15. RQ3: Quality of LLM-Inferred Taint Specifications
  16. ...and 21 more sections

Figures (11)

  • Figure 1: Overview of the IRIS neuro-symbolic system. It checks a given whole repository for a given type of vulnerability (CWE) and outputs a set of potential vulnerable paths with explanations.
  • Figure 2: An example of Code Injection (CWE-94) vulnerability found in cron-utils (CVE-2021-41269) that CodeQL fails to detect. We number the program points of the vulnerable path.
  • Figure 3: An illustration of the IRIS pipeline.
  • Figure 3: Ablation on LLM inferred source and sink specifications (CodeQL (QL) versus GPT-4), evaluated using the #Detected metrics. When replacing either source or sink with CodeQL specs, we see significantly less vulnerabilities detected.
  • Figure 4: LLM user prompt and response for contextual analysis of dataflow paths. In the user prompt, we mark with color the CWE and path information that is filling the prompt template. For cleaner presentation, we modify the snippets and left out the system prompt.
  • ...and 6 more figures