On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code

TL;DR

This study investigates how pre-trained language models of code cope with out-of-distribution data arising from software evolution, by pre-training from scratch on ID data and continually fine-tuning on five OOD domains introducing unseen APIs. It compares a GPT-2 decoder and a RoBERTa encoder on API call and API usage prediction tasks, demonstrating that naive fine-tuning suffers substantial forgetting, while replay-based and regularization-based continual learning methods substantially mitigate forgetting and achieve comparable or superior performance. The work contributes a large Java dataset with a well-controlled OOD split, a thorough evaluation of CL baselines in code PLMs, and evidence that continual learning is a promising direction for maintaining robustness to API evolution in code-related tasks. It also discusses threats to validity, broader impacts, and future directions, including benchmarks and potential applicability to LLMs with parameter-efficient fine-tuning.

Abstract

Pre-trained language models (PLMs) have become a prevalent technique in deep learning for code, utilizing a two-stage pre-training and fine-tuning procedure to acquire general knowledge about code and specialize in a variety of downstream tasks. However, the dynamic nature of software codebases poses a challenge to the effectiveness and robustness of PLMs. In particular, world-realistic scenarios potentially lead to significant differences between the distribution of the pre-training and test data, i.e., distribution shift, resulting in a degradation of the PLM's performance on downstream tasks. In this paper, we stress the need for adapting PLMs of code to software data whose distribution changes over time, a crucial problem that has been overlooked in previous works. The motivation of this work is to consider the PLM in a non-stationary environment, where fine-tuning data evolves over time according to a software evolution scenario. Specifically, we design a scenario where the model needs to learn from a stream of programs containing new, unseen APIs over time. We study two widely used PLM architectures, i.e., a GPT2 decoder and a RoBERTa encoder, on two downstream tasks, API call and API usage prediction. We demonstrate that the most commonly used fine-tuning technique from prior work is not robust enough to handle the dynamic nature of APIs, leading to the loss of previously acquired knowledge i.e., catastrophic forgetting. To address these issues, we implement five continual learning approaches, including replay-based and regularization-based methods. Our findings demonstrate that utilizing these straightforward methods effectively mitigates catastrophic forgetting in PLMs across both downstream tasks while achieving comparable or superior performance.

Figures (6)

• Figure 1: Continual fine-tuning of a pre-trained language model of code. After pre-training, the model needs to adapt to new out-of-distribution (OOD) program data over time.
• Figure 2: Procedure to extract the ID data used for model pre-training, and the OOD data used for continual fine-tuning.
• Figure 3: Overview of the downstream tasks. In the API call prediction task, the model outputs a list of top-$k$ candidates to predict the API call token (i.e., min). In the API usage prediction task, the model attempts to predict all the tokens constituting the API usage (interface name, method name, parameters and syntactical tokens). The models only leverage left-context tokens to generate a prediction.
• Figure 4: Naive fine-tuning approach results.
• Figure 5: Comparison of naive and cumulative fine-tuning settings for both models on API call prediction (EM@1).