Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Impact of AI-Assisted Development on Software Security: A Study of Gemini and Developer Experience

Nadine Jost, Benjamin Berens, Manuel Karl, Stefan Albert Horstmann, Martin Johns, Alena Naiakshina

Abstract

The ongoing shortage of skilled developers, particularly in security-critical software development, has led organizations to increasingly adopt AI-powered development tools to boost productivity and reduce reliance on limited human expertise. These tools, often based on large language models, aim to automate routine tasks and make secure software development more accessible and efficient. However, it remains unclear how developers' general programming and security-specific experience, and the type of AI tool used (free vs. paid) affect the security of the resulting software. Therefore, we conducted a quantitative programming study with software developers (n=159) exploring the impact of Google's AI tool Gemini on code security. Participants were assigned a security-related programming task using either no AI tools, the free version, or the paid version of Gemini. While we did not observe significant differences between using Gemini in terms of secure software development, programming experience significantly improved code security and cannot be fully substituted by Gemini.

The Impact of AI-Assisted Development on Software Security: A Study of Gemini and Developer Experience

Abstract

The ongoing shortage of skilled developers, particularly in security-critical software development, has led organizations to increasingly adopt AI-powered development tools to boost productivity and reduce reliance on limited human expertise. These tools, often based on large language models, aim to automate routine tasks and make secure software development more accessible and efficient. However, it remains unclear how developers' general programming and security-specific experience, and the type of AI tool used (free vs. paid) affect the security of the resulting software. Therefore, we conducted a quantitative programming study with software developers (n=159) exploring the impact of Google's AI tool Gemini on code security. Participants were assigned a security-related programming task using either no AI tools, the free version, or the paid version of Gemini. While we did not observe significant differences between using Gemini in terms of secure software development, programming experience significantly improved code security and cannot be fully substituted by Gemini.
Paper Structure (41 sections, 1 equation, 6 figures, 12 tables)

This paper contains 41 sections, 1 equation, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI Assistance Tool Studies
  4. Security Developer Studies
  5. Methodology
  6. Programming Task
  7. Pilot Study
  8. Power Analysis
  9. Participants
  10. Evaluation
  11. Methodology: Statistical Analysis and Hypothesis Testing
  12. Model diagnostics.
  13. Hypothesis testing.
  14. Effect‑size estimation.
  15. Model fit.
  16. ...and 26 more sections

Figures (6)

  • Figure 1: Correlation Programming Experience and Security Score.
  • Figure 2: Violinplot for Security Score Divided by Group (0 = lowest, 5 = highest security score).
  • Figure 3: Responses for *"I trust Gemini in general" and **"I trust Gemini to generate secure code".
  • Figure 4: Responses for *"Would you trust AI in general?" and **"Would you trust AI to generate secure code?", Asked for No-AI Group
  • Figure 5: Responses for "I believe I have solved this task functionally correctly."
  • ...and 1 more figures