Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating API Parameter Security Rules with LLM for API Misuse Detection

Jinghua Liu, Yi Yang, Kai Chen, Miaoqian Lin

TL;DR

GPTAid generates 6 times more APSRs than state-of-the-art detectors on a comparison dataset of previously reported bugs and APSRs and validates the correctness of the LLM-generated APSRs to validate the correctness of the LLM-generated APSRs.

Abstract

In this paper, we present a new framework, named GPTAid, for automatic APSRs generation by analyzing API source code with LLM and detecting API misuse caused by incorrect parameter use. To validate the correctness of the LLM-generated APSRs, we propose an execution feedback-checking approach based on the observation that security-critical API misuse is often caused by APSRs violations, and most of them result in runtime errors. Specifically, GPTAid first uses LLM to generate raw APSRs and the Right calling code, and then generates Violation code for each raw APSR by modifying the Right calling code using LLM. Subsequently, GPTAid performs dynamic execution on each piece of Violation code and further filters out the incorrect APSRs based on runtime errors. To further generate concrete APSRs, GPTAid employs a code differential analysis to refine the filtered ones. Particularly, as the programming language is more precise than natural language, GPTAid identifies the key operations within Violation code by differential analysis, and then generates the corresponding concrete APSR based on the aforementioned operations. These concrete APSRs could be precisely interpreted into applicable detection code, which proven to be effective in API misuse detection. Implementing on the dataset containing 200 randomly selected APIs from eight popular libraries, GPTAid achieves a precision of 92.3%. Moreover, it generates 6 times more APSRs than state-of-the-art detectors on a comparison dataset of previously reported bugs and APSRs. We further evaluated GPTAid on 47 applications, 210 unknown security bugs were found potentially resulting in severe security issues (e.g., system crashes), 150 of which have been confirmed by developers after our reports.

Generating API Parameter Security Rules with LLM for API Misuse Detection

TL;DR

GPTAid generates 6 times more APSRs than state-of-the-art detectors on a comparison dataset of previously reported bugs and APSRs and validates the correctness of the LLM-generated APSRs to validate the correctness of the LLM-generated APSRs.

Abstract

In this paper, we present a new framework, named GPTAid, for automatic APSRs generation by analyzing API source code with LLM and detecting API misuse caused by incorrect parameter use. To validate the correctness of the LLM-generated APSRs, we propose an execution feedback-checking approach based on the observation that security-critical API misuse is often caused by APSRs violations, and most of them result in runtime errors. Specifically, GPTAid first uses LLM to generate raw APSRs and the Right calling code, and then generates Violation code for each raw APSR by modifying the Right calling code using LLM. Subsequently, GPTAid performs dynamic execution on each piece of Violation code and further filters out the incorrect APSRs based on runtime errors. To further generate concrete APSRs, GPTAid employs a code differential analysis to refine the filtered ones. Particularly, as the programming language is more precise than natural language, GPTAid identifies the key operations within Violation code by differential analysis, and then generates the corresponding concrete APSR based on the aforementioned operations. These concrete APSRs could be precisely interpreted into applicable detection code, which proven to be effective in API misuse detection. Implementing on the dataset containing 200 randomly selected APIs from eight popular libraries, GPTAid achieves a precision of 92.3%. Moreover, it generates 6 times more APSRs than state-of-the-art detectors on a comparison dataset of previously reported bugs and APSRs. We further evaluated GPTAid on 47 applications, 210 unknown security bugs were found potentially resulting in severe security issues (e.g., system crashes), 150 of which have been confirmed by developers after our reports.
Paper Structure (33 sections, 15 figures, 10 tables)

This paper contains 33 sections, 15 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Background
  4. API Parameter Security Rules
  5. Large Language Model
  6. Methodology
  7. Overview
  8. Raw APSRs Generation
  9. APSRs Validation
  10. APSRs Refinement
  11. API misuse detection
  12. Implementation
  13. Evaluation
  14. Settings
  15. Effectiveness
  16. ...and 18 more sections

Figures (15)

  • Figure 1: Example for an API misuse in darktable
  • Figure 2: Overly general APSR leads to incorrect interpretations
  • Figure 3: Architecture of GPTAid
  • Figure 4: An example of GPTAid's workflow
  • Figure 5: Prompt example of Raw APSRs Generation
  • ...and 10 more figures