Time to Separate from StackOverflow and Match with ChatGPT for Encryption

TL;DR

The paper investigates how developers use StackOverflow to implement Java JCA symmetric encryption, revealing widespread security misuses in code examples and challenging practical cryptography practices. It combines manual coding, rule-based security analysis, and an evaluation of ChatGPT responses to quantify developer difficulties and AI-assisted potential. Key findings show dominant challenges in key/IV management and padding, pervasive security violations across posts, and that ChatGPT can help when guided but cannot replace expert human oversight. The work contributes a dataset of symmetric-encryption posts, a rule set for detecting misuses, and insights into AI-assisted code security, underscoring the need for caution when relying on community snippets or AI-driven code generation in cryptography-sensitive contexts.

Abstract

Cryptography is known as a challenging topic for developers. We studied StackOverflow posts to identify the problems that developers encounter when using Java Cryptography Architecture (JCA) for symmetric encryption. We investigated security risks that are disseminated in these posts, and we examined whether ChatGPT helps avoid cryptography issues. We found that developers frequently struggle with key and IV generations, as well as padding. Security is a top concern among developers, but security issues are pervasive in code snippets. ChatGPT can effectively aid developers when they engage with it properly. Nevertheless, it does not substitute human expertise, and developers should remain alert.

Figures (4)

• Figure 1: Distribution of symmetric encryption posts
• Figure 2: Overview of our research methodology
• Figure 3: Distribution of non-functional issues
• Figure 4: Share of encryption modes for AES