How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning
Fabio Salerno, Ali Al-Kaswan, Maliheh Izadi
TL;DR
This study investigates how much code language models remember, focusing on pre-training and fine-tuning data memorization. It introduces a data extraction security game and a custom benchmark to quantify extractability, evaluating StarCoder2 models across sizes using Java line completion tasks. Key findings show that pre-training data memorization exists and increases with model size and longer prefixes, while fine-tuning reduces extractability of pre-training data but amplifies vulnerability of fine-tuning data for smaller models; data carriers and licenses emerge as the most memorized data types, with licenses more likely to be forgotten after fine-tuning. The work also demonstrates that data duplication and input prefix length significantly influence leakage and provides replication artifacts to enable future research and mitigation strategies in code LLMs.
Abstract
Code language models, while widely popular, are often trained on unsanitized source code gathered from across the Internet. Previous work revealed that pre-trained models can remember the content of their training data and regurgitate them through data extraction attacks. Due to the large size of current models, only a few entities have the resources for pre-training such models. However, fine-tuning requires fewer resources and is increasingly used by both small and large entities for its effectiveness on specialized data. Such small curated data for fine-tuning might contain sensitive information or proprietary assets. In this study, we attack both pre-trained and fine-tuned code language models to investigate the extent of data extractability. We first develop a custom benchmark to assess the vulnerability of both pre-training and fine-tuning samples to extraction attacks. Our findings reveal that 54.9% of extractable pre-training data could be retrieved from StarCoder2-15B, whereas this number decreased to 23.5% after fine-tuning. This indicates that fine-tuning reduces the extractability of pre-training data. However, compared to larger models, fine-tuning smaller models increases their vulnerability to data extraction attacks on fine-tuning data. Given the potential sensitivity of fine-tuning data, this can lead to more severe consequences. Lastly, we also manually analyzed 2000 extractable samples before and after fine-tuning. We also found that data carriers and licensing information are the most likely data categories to be memorized from pre-trained and fine-tuned models, while the latter is the most likely to be forgotten after fine-tuning.