Secure Coding with AI, From Creation to Inspection

TL;DR

This study analyzes 1,586 real-world code snippets from the DevGPT dataset across C#, C++, and C to assess ChatGPT’s security of AI-generated code and its ability to detect and repair vulnerabilities. Using four static scanners, 124 files showed potential issues, and manual validation yielded 32 confirmed vulnerabilities in 26 files. When prompted to detect and fix issues, ChatGPT identified 18 of 32 vulnerabilities and fixed 17, but failed to recognize or correctly fix the rest, with 22 issues arising from ChatGPT-generated code and 10 from user prompts. The findings emphasize that ChatGPT is not yet reliable for secure code generation or comprehensive vulnerability detection, underscoring the ongoing importance of static analysis and human review in the software development lifecycle. The work also highlights the potential for AI tools to misinform with high confidence and the need for rigorous validation in AI-assisted security workflows.

Abstract

While prior studies have explored security in code generated by ChatGPT and other Large Language Models, they were conducted in controlled experimental settings and did not use code generated or provided from actual developer interactions. This paper not only examines the security of code generated by ChatGPT based on real developer interactions, curated in the DevGPT dataset, but also assesses ChatGPT's capability to find and fix these vulnerabilities. We analysed 1,586 C, C++, and C# code snippets using static scanners, which detected potential issues in 124 files. After manual analysis, we selected 26 files with 32 confirmed vulnerabilities for further investigation. We submitted these files to ChatGPT via the OpenAI API, asking it to detect security issues, identify the corresponding Common Weakness Enumeration numbers, and propose fixes. The responses and modified code were manually reviewed and re-scanned for vulnerabilities. ChatGPT successfully detected 18 out of 32 security issues and resolved 17 issues but failed to recognize or fix the remainder. Interestingly, only 10 vulnerabilities were resulted from the user prompts, while 22 were introduced by ChatGPT itself. We highlight for developers that code generated by ChatGPT is more likely to contain vulnerabilities compared to their own code. Furthermore, at times ChatGPT reports incorrect information with apparent confidence, which may mislead less experienced developers. Our findings confirm previous studies in demonstrating that ChatGPT is not sufficiently reliable for generating secure code nor identifying all vulnerabilities, highlighting the continuing importance of static scanners and manual review.