How I Learned to Stop Worrying and Love ChatGPT
Piotr Przymus, Mikołaj Fejzer, Jakub Narębski, Krzysztof Stencel
TL;DR
The study investigates how ChatGPT interactions influence software development by linking ChatGPT conversations to code changes across commits, pull requests, and issues using the DevGPT dataset. It uses Gestalt pattern matching to quantify how much the code was derived from or inspired by ChatGPT prompts, and applies Kaplan-Meier survival analysis to compare the longevity of ChatGPT-influenced code against human-written code. Key findings show that commits show stronger direct ChatGPT influence while issues and pull requests serve more for ideation, and that ChatGPT-inspired code tends to undergo changes more rapidly, likely due to limited project context and iterative refinement. These insights illuminate the nuanced role of AI-assisted coding in code evolution and maintenance, with implications for responsible usage and tooling aroundChatGPT-generated code.
Abstract
In the dynamic landscape of software engineering, the emergence of ChatGPT-generated code signifies a distinctive and evolving paradigm in development practices. We delve into the impact of interactions with ChatGPT on the software development process, specifically analysing its influence on source code changes. Our emphasis lies in aligning code with ChatGPT conversations, separately analysing the user-provided context of the code and the extent to which the resulting code has been influenced by ChatGPT. Additionally, employing survival analysis techniques, we examine the longevity of ChatGPT-generated code segments in comparison to lines written traditionally. The goal is to provide valuable insights into the transformative role of ChatGPT in software development, illuminating its implications for code evolution and sustainability within the ecosystem.