The role of library versions in Developer-ChatGPT conversations

TL;DR

This work investigates whether and how library versions influence Developer-ChatGPT conversations. Using the DevGPT dataset of over 4k interactions, the authors combine quantitative filtering and qualitative coding to quantify the rarity of explicit library-version mentions (about 9.7%) and to characterize how users and ChatGPT handle version information, including scenarios of configuration, troubleshooting, and pinning. The findings suggest that most exchanges omit version constraints, while when versions are discussed, they largely reflect user-driven prompts or proactive guidance from ChatGPT in setup tasks. These results highlight the need for more robust, version-aware guidance in code-generation assistants and motivate further study on API compatibility across library versions. The work provides a data-driven baseline for understanding version-awareness in developer-facing AI tools and points to practical questions for improving resilience to library evolution in code recommendations.

Abstract

The latest breakthroughs in large language models (LLM) have empowered software development tools, such as ChatGPT, to aid developers in complex tasks. Developers use ChatGPT to write code, review code changes, and even debug their programs. In these interactions, ChatGPT often recommends code snippets that depend on external libraries. However, code from libraries changes over time, invalidating a once-correct code snippet and making it difficult to reuse recommended code. In this study, we analyze DevGPT, a dataset of more than 4,000 Developer-ChatGPT interactions, to understand the role of library versions in code-related conversations. We quantify how often library version constraints are mentioned in code-related conversations and when ChatGPT recommends the installation of specific libraries. Our findings show that, albeit to constantly recommend and analyze code with external dependencies, library version constraints only appear in 9% of the conversations. In the majority of conversations, the version constraints are prompted by users (as opposed to being specified by ChatGPT) as a method for receiving better quality responses. Moreover, we study how library version constraints are used in the conversation through qualitative methods, identifying several potential problems that warrant further research.