Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Memorization in Fine-tuned Language Models

Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, Dawei Yin

TL;DR

This first comprehensive analysis to explore language models' memorization during fine-tuning across tasks indicates that memorization presents a strong disparity among different fine-tuning tasks and provides an intuitive explanation of this task disparity via sparse coding theory.

Abstract

Large language models (LLMs) have shown great capabilities in various tasks but also exhibited memorization of training data, raising tremendous privacy and copyright concerns. While prior works have studied memorization during pre-training, the exploration of memorization during fine-tuning is rather limited. Compared to pre-training, fine-tuning typically involves more sensitive data and diverse objectives, thus may bring distinct privacy risks and unique memorization behaviors. In this work, we conduct the first comprehensive analysis to explore language models' (LMs) memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.

Exploring Memorization in Fine-tuned Language Models

TL;DR

This first comprehensive analysis to explore language models' memorization during fine-tuning across tasks indicates that memorization presents a strong disparity among different fine-tuning tasks and provides an intuitive explanation of this task disparity via sparse coding theory.

Abstract

Large language models (LLMs) have shown great capabilities in various tasks but also exhibited memorization of training data, raising tremendous privacy and copyright concerns. While prior works have studied memorization during pre-training, the exploration of memorization during fine-tuning is rather limited. Compared to pre-training, fine-tuning typically involves more sensitive data and diverse objectives, thus may bring distinct privacy risks and unique memorization behaviors. In this work, we conduct the first comprehensive analysis to explore language models' (LMs) memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.
Paper Structure (54 sections, 5 equations, 14 figures, 16 tables)

This paper contains 54 sections, 5 equations, 14 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Definitions and Notations
  5. Definitions of memorization in literature.
  6. Definition of fine-tuned memorization.
  7. Evaluation method.
  8. Preliminary Findings
  9. Disparate Memorization Across Tasks
  10. Fine-tuned with Fixed Pre-trained LM and Dataset
  11. Fine-tuned on T5-base LM.
  12. Fine-tuned on different tasks with the same dataset.
  13. Further Probing
  14. Input prefix length.
  15. Generation sampling.
  16. ...and 39 more sections

Figures (14)

  • Figure 1: Memorization of T5-base fine-tuned on RentTheRunway.
  • Figure 2: Impact of prefix length and sampling methods on memorization
  • Figure 3: Scaling behavior of fine-tuned memorization
  • Figure 4: Decoder-encoder attention heatmaps for (a, b) Summarization, (c, d) Dialog, (e, f) Sentiment Analysis, and (g, h) QA. (a, c, e, g) show average heatmaps from 10 samples, while (b, d, f, h) show heatmaps from a single sample.
  • Figure 5: Decoder-encoder attention heatmaps on translation.
  • ...and 9 more figures

Theorems & Definitions (1)

  • Definition 1: Fine-tuned memorization