Investigating the Utility of ChatGPT in the Issue Tracking System: An Exploratory Study
Joy Krishan Das, Saikat Mondal, Chanchal K. Roy
TL;DR
Open-source issue trackers face high workload and limited contributor bandwidth, prompting investigation into how developers actually use ChatGPT for issue resolution. The authors analyze the DevGPT dataset with qualitative coding and NiCad clone detection to characterize usage patterns and code integration, finding that ChatGPT mainly serves as an ideation and guidance partner, while direct integration of its code into projects is rare. Feature-focused code generation occurs in some cases, but concerns about hallucinations and reliability reduce adoption of generated code. The study highlights the practical value of AI for brainstorming and planning in issue resolution and underscores the need for safer, human-in-the-loop workflows to harness AI effectively.
Abstract
Issue tracking systems serve as the primary tool for incorporating external users and customizing a software project to meet the users' requirements. However, the limited number of contributors and the challenge of identifying the best approach for each issue often impede effective resolution. Recently, an increasing number of developers are turning to AI tools like ChatGPT to enhance problem-solving efficiency. While previous studies have demonstrated the potential of ChatGPT in areas such as automatic program repair, debugging, and code generation, there is a lack of study on how developers explicitly utilize ChatGPT to resolve issues in their tracking system. Hence, this study aims to examine the interaction between ChatGPT and developers to analyze their prevalent activities and provide a resolution. In addition, we assess the code reliability by confirming if the code produced by ChatGPT was integrated into the project's codebase using the clone detection tool NiCad. Our investigation reveals that developers mainly use ChatGPT for brainstorming solutions but often opt to write their code instead of using ChatGPT-generated code, possibly due to concerns over the generation of "hallucinated code", as highlighted in the literature.