Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The potential of LLM-generated reports in DevSecOps

Nikolaos Lykousas, Vasileios Argyropoulos, Fran Casino

TL;DR

This paper addresses alert fatigue in DevSecOps by exploring LLM-generated reports that summarize security findings with a focus on financial impact. The authors use LAZARUS AI Secret Scanner, which relies on two TextCNN models to identify hardcoded credentials, to generate inputs for LLMs (ChatGPT and Llama 3) that produce actionable risk reports. A survey with $N=23$ developers evaluating $20$ reports shows that LLM-generated reports can improve perceived clarity and motivation to remediate compared with standard alerts, though trust and cost-accuracy issues persist, including hallucinated financial figures. The work suggests integrating LLM-based reporting into DevSecOps workflows, with potential advantages for on-premises, open-source models to address trust concerns.

Abstract

Alert fatigue is a common issue faced by software teams using the DevSecOps paradigm. The overwhelming number of warnings and alerts generated by security and code scanning tools, particularly in smaller teams where resources are limited, leads to desensitization and diminished responsiveness to security warnings, potentially exposing systems to vulnerabilities. This paper explores the potential of LLMs in generating actionable security reports that emphasize the financial impact and consequences of detected security issues, such as credential leaks, if they remain unaddressed. A survey conducted among developers indicates that LLM-generated reports significantly enhance the likelihood of immediate action on security issues by providing clear, comprehensive, and motivating insights. Integrating these reports into DevSecOps workflows can mitigate attention saturation and alert fatigue, ensuring that critical security warnings are addressed effectively.

The potential of LLM-generated reports in DevSecOps

TL;DR

This paper addresses alert fatigue in DevSecOps by exploring LLM-generated reports that summarize security findings with a focus on financial impact. The authors use LAZARUS AI Secret Scanner, which relies on two TextCNN models to identify hardcoded credentials, to generate inputs for LLMs (ChatGPT and Llama 3) that produce actionable risk reports. A survey with developers evaluating reports shows that LLM-generated reports can improve perceived clarity and motivation to remediate compared with standard alerts, though trust and cost-accuracy issues persist, including hallucinated financial figures. The work suggests integrating LLM-based reporting into DevSecOps workflows, with potential advantages for on-premises, open-source models to address trust concerns.

Abstract

Alert fatigue is a common issue faced by software teams using the DevSecOps paradigm. The overwhelming number of warnings and alerts generated by security and code scanning tools, particularly in smaller teams where resources are limited, leads to desensitization and diminished responsiveness to security warnings, potentially exposing systems to vulnerabilities. This paper explores the potential of LLMs in generating actionable security reports that emphasize the financial impact and consequences of detected security issues, such as credential leaks, if they remain unaddressed. A survey conducted among developers indicates that LLM-generated reports significantly enhance the likelihood of immediate action on security issues by providing clear, comprehensive, and motivating insights. Integrating these reports into DevSecOps workflows can mitigate attention saturation and alert fatigue, ensuring that critical security warnings are addressed effectively.
Paper Structure (9 sections, 3 figures, 2 tables)

This paper contains 9 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. DesvSecOps
  4. LLM Report Generation
  5. Methodology
  6. Survey Design and Results
  7. Conclusions
  8. Acknowledgments.
  9. Disclosure of Interests.

Figures (3)

  • Figure 1: Example source code file with hardcoded credentials, and the corresponding output of LAZARUS AI Secret Scanner.
  • Figure 2: LLM-generated reports
  • Figure 3: Answers to the survey per question and LLM in Likert-scale.