Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions

Amirali Sajadi, Binh Le, Anh Nguyen, Kostadin Damevski, Preetha Chatterjee

TL;DR

This study investigates whether contemporary LLMs exhibit security awareness when answering programming questions. By prompting GPT-4, Claude 3, and Llama 3 with 300 Stack Overflow questions containing vulnerable code (split into Mentions-Dataset and Transformed-Dataset), the authors measure whether models warn users about vulnerabilities and whether their explanations cover causes, exploits, and fixes. Findings show a generally low detection rate (12.6%–40%), with GPT-4 outperforming peers and higher warning rates when the SO responses already mention security. When warnings occur, LLMs tend to provide more information about causes and fixes than typical SO responses, and exploiting information is strongest in GPT-4 outputs. The work further demonstrates that prompt engineering and CodeQL-based tool integration can significantly enhance security-aware responses, and it discusses implications for software engineers, SE tool integration, and LLM design—highlighting a need for standardized evaluation and domain-specific training to make LLM-assisted programming safer in practice.

Abstract

The widespread adoption of conversational LLMs for software development has raised new security concerns regarding the safety of LLM-generated content. Our motivational study outlines ChatGPT's potential in volunteering context-specific information to the developers, promoting safe coding practices. Motivated by this finding, we conduct a study to evaluate the degree of security awareness exhibited by three prominent LLMs: Claude 3, GPT-4, and Llama 3. We prompt these LLMs with Stack Overflow questions that contain vulnerable code to evaluate whether they merely provide answers to the questions or if they also warn users about the insecure code, thereby demonstrating a degree of security awareness. Further, we assess whether LLM responses provide information about the causes, exploits, and the potential fixes of the vulnerability, to help raise users' awareness. Our findings show that all three models struggle to accurately detect and warn users about vulnerabilities, achieving a detection rate of only 12.6% to 40% across our datasets. We also observe that the LLMs tend to identify certain types of vulnerabilities related to sensitive information exposure and improper input neutralization much more frequently than other types, such as those involving external control of file names or paths. Furthermore, when LLMs do issue security warnings, they often provide more information on the causes, exploits, and fixes of vulnerabilities compared to Stack Overflow responses. Finally, we provide an in-depth discussion on the implications of our findings and present a CLI-based prompting tool that can be used to generate significantly more secure LLM responses.

Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions

TL;DR

This study investigates whether contemporary LLMs exhibit security awareness when answering programming questions. By prompting GPT-4, Claude 3, and Llama 3 with 300 Stack Overflow questions containing vulnerable code (split into Mentions-Dataset and Transformed-Dataset), the authors measure whether models warn users about vulnerabilities and whether their explanations cover causes, exploits, and fixes. Findings show a generally low detection rate (12.6%–40%), with GPT-4 outperforming peers and higher warning rates when the SO responses already mention security. When warnings occur, LLMs tend to provide more information about causes and fixes than typical SO responses, and exploiting information is strongest in GPT-4 outputs. The work further demonstrates that prompt engineering and CodeQL-based tool integration can significantly enhance security-aware responses, and it discusses implications for software engineers, SE tool integration, and LLM design—highlighting a need for standardized evaluation and domain-specific training to make LLM-assisted programming safer in practice.

Abstract

The widespread adoption of conversational LLMs for software development has raised new security concerns regarding the safety of LLM-generated content. Our motivational study outlines ChatGPT's potential in volunteering context-specific information to the developers, promoting safe coding practices. Motivated by this finding, we conduct a study to evaluate the degree of security awareness exhibited by three prominent LLMs: Claude 3, GPT-4, and Llama 3. We prompt these LLMs with Stack Overflow questions that contain vulnerable code to evaluate whether they merely provide answers to the questions or if they also warn users about the insecure code, thereby demonstrating a degree of security awareness. Further, we assess whether LLM responses provide information about the causes, exploits, and the potential fixes of the vulnerability, to help raise users' awareness. Our findings show that all three models struggle to accurately detect and warn users about vulnerabilities, achieving a detection rate of only 12.6% to 40% across our datasets. We also observe that the LLMs tend to identify certain types of vulnerabilities related to sensitive information exposure and improper input neutralization much more frequently than other types, such as those involving external control of file names or paths. Furthermore, when LLMs do issue security warnings, they often provide more information on the causes, exploits, and fixes of vulnerabilities compared to Stack Overflow responses. Finally, we provide an in-depth discussion on the implications of our findings and present a CLI-based prompting tool that can be used to generate significantly more secure LLM responses.

Paper Structure

This paper contains 17 sections, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivational Study
  3. Methodology
  4. Dataset Collection and Refactoring
  5. Experimental Setup
  6. Results
  7. RQ1: Given insecure code, do LLMs warn developers of the potential security flaws or required security modifications?
  8. RQ2: In instances where users are reminded of security concerns, are they informed about the causes, potential exploits, and possible fixes of the vulnerabilities?
  9. Implications
  10. Implications for Software Engineers
  11. Implications for Tool Integration in Software Development
  12. Implications for LLM Designers
  13. Threats to Validity
  14. Related Work
  15. Security Concerns with using LLMs
  16. ...and 2 more sections

Figures (10)

  • Figure 1: DevGPT: ChatGPT Reminding User about the Security of the Code.
  • Figure 2: Overview of our study methodology.
  • Figure 3: Example of a SO Question Containing Vulnerable Code (Highlighted Red), and a User Response (Comment), Pointing Out the Vulnerability.
  • Figure 4: Overlap of the Identified Vulnerable Questions by the Three LLMs
  • Figure 5: 3 LLMs Point Out the Security Vulnerability in a Question/Prompt.
  • ...and 5 more figures