Unveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub
Ruiyin Li, Peng Liang, Yifei Wang, Yangxiao Cai, Weisong Sun, Zengyang Li
TL;DR
This study empirically analyzes how developers use ChatGPT in real-world software development by examining publicly shared GitHub conversations. It introduces DevChat, a dataset of 2,547 ChatGPT links across Code, Commits, Issues, Pull Requests, and Discussions (May 2023–June 2024) and uses grounded theory to derive five developer purposes, nine development-related activities, and 39 SE tasks, with Code Generation & Completion and Code Modification & Optimization as the most prominent tasks. The results reveal predominantly short, task-focused interactions and a strong bias toward coding and maintenance activities, while early-stage activities like requirements elicitation are less represented. The work provides a mapping framework linking data sources, activities, and SE tasks, offers practical guidance for integrating AI into development workflows, and highlights research opportunities to extend GenAI use into underexplored SE areas with attention to reliability and safety.
Abstract
The advent of Large Language Models (LLMs) has introduced a new paradigm in Software Engineering (SE), with generative AI tools like ChatGPT gaining widespread adoption among developers. While ChatGPT's potential has been extensively discussed, empirical evidence about how developers actually use LLMs' assistance in real-world practices remains limited. To bridge this gap, we conducted a large-scale empirical analysis of ChatGPT usage on GitHub, and we presented DevChat, a curated dataset of 2,547 publicly shared ChatGPT conversation links collected from GitHub between May 2023 and June 2024. Through comprehensively analyzing DevChat, we explored the characteristics of developer-ChatGPT interaction patterns and identified five key categories of developers' purposes for sharing developer-ChatGPT conversations during software development. Additionally, we investigated the dominant development-related activities in which ChatGPT is used, and presented a mapping framework that links GitHub data sources, development-related activities, and SE tasks. The findings show that interactions are typically short and task-focused (most are 1-3 turns); developers share conversations mainly to delegate tasks, resolve problems, and acquire knowledge, revealing five purpose categories; ChatGPT is most frequently engaged for Software Implementation and Maintenance & Evolution; we identified 39 fine-grained SE tasks supported by ChatGPT, with Code Generation & Completion as well as Code modification & Optimization being the most prominent. Our study offers a comprehensive mapping of ChatGPT's applications in real-world software development scenarios and provides a foundation for understanding LLMs' practical roles in software development.