Early Detection and Reduction of Memorisation for Domain Adaptation and Instruction Tuning

TL;DR

This paper examines how fine-tuning domain adaptation and instruction tuning of large language models leads to memorisation of training data, often early in training. It introduces an $n$-gram memorisation score as a practical early-warning signal and shows that early stopping based on this signal can reduce memorisation with minimal performance loss, while an $n$-gram regularisation loss further lowers memorisation by up to around 40% across models. The study evaluates multiple model families (e.g., Pythia, Llama3, Mistral) and diverse datasets, finding that memorisation scales with model size and that mitigation strategies vary in effectiveness depending on architecture and task. Overall, the work provides actionable, scalable insights into memorisation dynamics and practical defenses during fine-tuning in domain adaptation and instruction-tuning contexts.

Abstract

Although large language models excel across many tasks, they can memorise training data and thereby expose private or copyrighted text. Most defences target the pre-training stage, leaving memorisation during fine-tuning, especially for domain adaptation and instruction tuning, poorly understood. We fine-tune Pythia, Llama3, and Mistral models spanning 1.4B-70B parameters on common evaluation datasets and track verbatim memorisation throughout training. We find that memorisation increases dramatically in the first few epochs, often significantly before either validation perplexity or evaluation performance is optimised. We use a simple but effective n-gram memorisation score which reliably precedes verbatim memorisation; using it as an early-stopping criterion mitigates memorisation with minimal performance loss. Further, we introduce an n-gram-aware loss regulariser and show that it reduces memorisation across all model families tested by up to 40% while minimising evaluation performance trade-offs when compared to an existing memorisation mitigation strategy. These results yield practical, scalable insights into memorisation dynamics during language model fine-tuning.

Figures (6)

• Figure 1: Memorisation increases (number of new samples memorised at a given epoch) at successive fine-tuning epochs, comparing fine-tuning for domain adaptation (orange) and instruction tuning (purple) on the same data. Dashed vertical lines mark the average epoch for which validation perplexity (orange) and task evaluation accuracy (purple) are achieved, showing high memorisation before for both (a) Pythia 1.4B and (b) Pythia 12B models.
• Figure 2: Partial $n$-gram memorisation across fine-tuning epochs for the four dataset categories, with domain indicating domain adaptation fine-tuning. In each panel the coloured solid line reports, at epoch $t$, the median score of samples that become memorised at subsequent epoch $t{+}1$; the point colour encoding that memorisation epoch. The grey shaded region spans the full score range of all samples that ever become memorised, irrespective of when the transition occurs. Error bars show the standard deviation over five random seeds, while the black dashed line is the baseline for samples that are never memorised. Results are averages over Pythia model sizes from 1.4 B to 12 B parameters.
• Figure 3: Partial memorisation across epochs for different models. The coloured solid line gives the median score of samples that will be memorised at epoch $t{+}1$; point colour marks that future epoch. The grey region shows the full range for all eventually memorised samples, while the black dashed line is the baseline for samples never memorised. Error bars denote the standard deviation across five random seeds.
• Figure 4: Final-layer partial fine-tuning comparison of the Llama3 8B model. The final $n$ layers of the model are unfrozen and updated when fine-tuning, with the remaining layers frozen.
• Figure 5: Memorisation and performance comparison for domain adaptation and instruction tuning across different early stopping selection criteria. (a) and (c) show the verbatim memorisation percentage for different values of extraction prompt prefix length $k\in\{12,16,20\}$ using three early stopping selection criteria: validation perplexity (Best val), evaluation performance (Best acc), and $n$-gram memorisation (Best $n$-gram) for domain adaptation (solid) and instruction tuning (hatched). (b) and (d) present the difference to the best task evaluation performance (orange) and validation perplexity (green), across the same selection criteria and fine-tuning approaches.