Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations
Jasmine Latendresse, SayedHassan Khatoonabadi, Ahmad Abdellatif, Emad Shihab
TL;DR
The paper investigates whether ChatGPT can act effectively as a software librarian by evaluating libraries it recommends in GPT-3.5 Turbo-generated Python code for real Stack Overflow prompts. It finds that ChatGPT favors third-party, mature libraries with generally permissive licenses, but a non-trivial fraction involve copyleft licenses or unspecified licenses, and about 6.5% of recommendations fail to import or install due to issues like aliases and ambiguous imports. The authors propose practical steps to mitigate risks, including explicit licensing and maintainability data in responses, dependency-footprint notes, and verification tooling, while acknowledging limitations such as potential training-data overlap and language-specific focus. Overall, the study presents a nuanced view: ChatGPT can function as a useful software librarian with targeted improvements to licensing transparency, dependency awareness, and out-of-the-box reliability.
Abstract
Software libraries play a critical role in the functionality, efficiency, and maintainability of software systems. As developers increasingly rely on Large Language Models (LLMs) to streamline their coding processes, the effectiveness of these models in recommending appropriate libraries becomes crucial yet remains largely unexplored. In this paper, we assess the effectiveness of ChatGPT as a software librarian and identify areas for improvement. We conducted an empirical study using GPT-3.5 Turbo to generate Python code for 10,000 Stack Overflow questions. Our findings show that ChatGPT uses third-party libraries nearly 10% more often than human developers, favoring widely adopted and well-established options. However, 14.2% of the recommended libraries had restrictive copyleft licenses, which were not explicitly communicated by ChatGPT. Additionally, 6.5% of the libraries did not work out of the box, leading to potential developer confusion and wasted time. While ChatGPT can be an effective software librarian, it should be improved by providing more explicit information on maintainability metrics and licensing. We recommend that developers implement rigorous dependency management practices and double-check library licenses before integrating LLM-generated code into their projects.