Exploring the Problems, their Causes and Solutions of AI Pair Programming: A Study on GitHub and Stack Overflow
Xiyu Zhou, Peng Liang, Beiqi Zhang, Zengyang Li, Aakash Ahmad, Mojtaba Shahin, Muhammad Waseem
TL;DR
This study systematically characterizes the problems, causes, and potential solutions encountered by developers using GitHub Copilot by mining GitHub Issues, GitHub Discussions, and Stack Overflow posts posted after Copilot's launch. Using open coding and constant comparison within a grounded theory framework, the authors identify 1,355 Copilot-related problems, 391 causal types, and 497 actionable solutions, revealing that Operation Issues and Compatibility Issues are the most frequent challenges, with Copilot Internal Error and Network Connection Error as leading causes. The work maps problems to causes and solutions, showing that server-side fixes and user-side configuration adjustments are the most common remediation paths, while feature requests and UX concerns often require new releases or broader ecosystem support. Practical implications are discussed for Copilot users (careful review of suggestions, choosing supported IDEs), the Copilot team (enhanced customization, broader IDE support, IP considerations), and researchers (value of code explanations and monitoring feature-request trends). Overall, the findings inform targeted improvements to Copilot’s reliability, usability, and governance, and suggest directions for future empirical studies and industrial evaluations.
Abstract
With the recent advancement of Artificial Intelligence (AI) and Large Language Models (LLMs), AI-based code generation tools become a practical solution for software development. GitHub Copilot, the AI pair programmer, utilizes machine learning models trained on a large corpus of code snippets to generate code suggestions using natural language processing. Despite its popularity in software development, there is limited empirical evidence on the actual experiences of practitioners who work with Copilot. To this end, we conducted an empirical study to understand the problems that practitioners face when using Copilot, as well as their underlying causes and potential solutions. We collected data from 473 GitHub issues, 706 GitHub discussions, and 142 Stack Overflow posts. Our results reveal that (1) Operation Issue and Compatibility Issue are the most common problems faced by Copilot users, (2) Copilot Internal Error, Network Connection Error, and Editor/IDE Compatibility Issue are identified as the most frequent causes, and (3) Bug Fixed by Copilot, Modify Configuration/Setting, and Use Suitable Version are the predominant solutions. Based on the results, we discuss the potential areas of Copilot for enhancement, and provide the implications for the Copilot users, the Copilot team, and researchers.