Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion

Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, Xin Peng

TL;DR

This work studies deprecated API usage in LLM-based code completion amid rapid library evolution. It builds a study with 145 Python API mappings, 9{,}022 outdated functions, and 19{,}103 up-to-dated functions across eight libraries, evaluated on seven LLMs, and introduces two lightweight fixes (REPLACEAPI and INSERTPROMPT) to mitigate deprecated usage. Key findings show that LLMs struggle to predict plausible APIs and exhibit substantial deprecated usage, influenced by model type, prompt design, and library characteristics; training data and lack of post-deprecation knowledge are major root causes. The paper demonstrates that ReplaceAPI reliably fixes deprecated usage with high success rates, while InsertPrompt shows potential but is less consistently effective, and discusses implications for validating, updating, and integrating library evolution with LLM-driven software development.

Abstract

Large language models (LLMs), pre-trained or fine-tuned on large code corpora, have shown effectiveness in generating code completions. However, in LLM-based code completion, LLMs may struggle to use correct and up-to-date Application Programming Interfaces (APIs) due to the rapid and continuous evolution of libraries. While existing studies have highlighted issues with predicting incorrect APIs, the specific problem of deprecated API usage in LLM-based code completion has not been thoroughly investigated. To address this gap, we conducted the first evaluation study on deprecated API usage in LLM-based code completion. This study involved seven advanced LLMs, 145 API mappings from eight popular Python libraries, and 28,125 completion prompts. The study results reveal the status quo (i.e., API usage plausibility and deprecated usage rate) of deprecated API and replacing API usage in LLM-based code completion from the perspectives of model, prompt, and library, and indicate the root causes behind. Based on these findings, we propose two lightweight fixing approaches, REPLACEAPI and INSERTPROMPT, which can serve as baseline approaches for future research on mitigating deprecated API usage in LLM-based completion. Additionally, we provide implications for future research on integrating library evolution with LLM-driven software development.

LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion

TL;DR

This work studies deprecated API usage in LLM-based code completion amid rapid library evolution. It builds a study with 145 Python API mappings, 9{,}022 outdated functions, and 19{,}103 up-to-dated functions across eight libraries, evaluated on seven LLMs, and introduces two lightweight fixes (REPLACEAPI and INSERTPROMPT) to mitigate deprecated usage. Key findings show that LLMs struggle to predict plausible APIs and exhibit substantial deprecated usage, influenced by model type, prompt design, and library characteristics; training data and lack of post-deprecation knowledge are major root causes. The paper demonstrates that ReplaceAPI reliably fixes deprecated usage with high success rates, while InsertPrompt shows potential but is less consistently effective, and discusses implications for validating, updating, and integrating library evolution with LLM-driven software development.

Abstract

Large language models (LLMs), pre-trained or fine-tuned on large code corpora, have shown effectiveness in generating code completions. However, in LLM-based code completion, LLMs may struggle to use correct and up-to-date Application Programming Interfaces (APIs) due to the rapid and continuous evolution of libraries. While existing studies have highlighted issues with predicting incorrect APIs, the specific problem of deprecated API usage in LLM-based code completion has not been thoroughly investigated. To address this gap, we conducted the first evaluation study on deprecated API usage in LLM-based code completion. This study involved seven advanced LLMs, 145 API mappings from eight popular Python libraries, and 28,125 completion prompts. The study results reveal the status quo (i.e., API usage plausibility and deprecated usage rate) of deprecated API and replacing API usage in LLM-based code completion from the perspectives of model, prompt, and library, and indicate the root causes behind. Based on these findings, we propose two lightweight fixing approaches, REPLACEAPI and INSERTPROMPT, which can serve as baseline approaches for future research on mitigating deprecated API usage in LLM-based completion. Additionally, we provide implications for future research on integrating library evolution with LLM-driven software development.
Paper Structure (32 sections, 5 equations, 5 figures, 5 tables, 3 algorithms)

This paper contains 32 sections, 5 equations, 5 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Library Evolution
  4. LLM-based Code Completion
  5. Study Setup
  6. API Mapping Collection
  7. Completion Prompt Construction
  8. Outdated and Up-to-dated Function Location.
  9. Incomplete Code Extraction.
  10. LLM-based Code Completion
  11. Completion Result Annotation
  12. Metrics
  13. Study Results
  14. RQ1: Model Perspective for Status Quo and Root Causes
  15. Status Quo Analysis
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Overview of Our Study
  • Figure 2: Illustration of Completion Prompt Construction for An Up-to-dated Function.
  • Figure 3: Distribution of RepC, DepC, and Others Completions by LLMs for Prompts from $\mathcal{O}$ and $\mathcal{U}$
  • Figure 4: AUP by Different LLMs across Eight Libraries
  • Figure 5: DUR by Different LLMs across Eight Libraries