Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study
Rosalia Tufano, Antonio Mastropaolo, Federica Pepe, Ozren Dabić, Massimiliano Di Penta, Gabriele Bavota
TL;DR
This study addresses the gap in empirical evidence on how developers actually use ChatGPT in open-source software. By mining 1,501 GitHub artifacts that mention ChatGPT and performing open coding, the authors derive a taxonomy of 45 tasks grouped into seven root categories, revealing predominant use in feature implementation, software quality, and documentation, among others. The findings highlight practical implications and risks, including ownership concerns, potential hallucinations, and the need for careful integration with existing development processes. The work provides a replication package and lays groundwork for future validated studies across additional LLMs and contexts, informing practitioners and guiding research on AI-assisted software development.
Abstract
Large Language Models (LLMs) have gained significant attention in the software engineering community. Nowadays developers have the possibility to exploit these models through industrial-grade tools providing a handy interface toward LLMs, such as OpenAI's ChatGPT. While the potential of LLMs in assisting developers across several tasks has been documented in the literature, there is a lack of empirical evidence mapping the actual usage of LLMs in software projects. In this work, we aim at filling such a gap. First, we mine 1,501 commits, pull requests (PRs), and issues from open-source projects by matching regular expressions likely to indicate the usage of ChatGPT to accomplish the task. Then, we manually analyze these instances, discarding false positives (i.e., instances in which ChatGPT was mentioned but not actually used) and categorizing the task automated in the 467 true positive instances (165 commits, 159 PRs, 143 issues). This resulted in a taxonomy of 45 tasks which developers automate via ChatGPT. The taxonomy, accompanied with representative examples, provides (i) developers with valuable insights on how to exploit LLMs in their workflow and (ii) researchers with a clear overview of tasks that, according to developers, could benefit from automated solutions.