How new data permeates LLM knowledge and how to dilute it

TL;DR

This work analyzes how new information inserted into LLMs reshapes prior knowledge, introducing the priming phenomenon where a learned fact influences unrelated contexts. By building Outlandish, a 1320-sample dataset with 12 keywords, the authors show that post-learning priming can be predicted from pre-learning keyword probability, a finding that holds across model families and training stages. They further demonstrate two practical mitigations—a stepping-stone text augmentation and an ignore-topk gradient pruning method—that substantially reduce priming (by up to 2 orders of magnitude) while preserving the model’s ability to learn new information. The results offer empirical insight into data-driven continual learning and provide actionable tools to improve the specificity of knowledge insertion in LLMs, with implications for safety and reliability in dynamic training regimes.

Abstract

Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hallucination, remains poorly understood. We demonstrate that when learning new information, LLMs exhibit a "priming" effect: learning a new fact can cause the model to inappropriately apply that knowledge in unrelated contexts. To systematically study this phenomenon, we introduce "Outlandish," a carefully curated dataset of 1320 diverse text samples designed to probe how new knowledge permeates through an LLM's existing knowledge base. Using this dataset, we show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before learning. This relationship holds robustly across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages. Finally, we develop two novel techniques to modulate how new knowledge affects existing model behavior: (1) a ``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update pruning method. These approaches reduce undesirable priming effects by 50-95\% while preserving the model's ability to learn new information. Our findings provide both empirical insights into how LLMs learn and practical tools for improving the specificity of knowledge insertion in language models. Further materials: https://sunchipsster1.github.io/projects/outlandish/

Figures (30)

• Figure 1: Outlandish dataset and main finding on "priming". (a) Sample texts within the Outlandish dataset. (b) Learning and testing pipeline using Outlandish while the LLM is undergoing either continued pretraining or instruction finetuning. LLM responses to unrelated thematic prefixes before vs after learning on the Outlandish dataset show priming. (c) The degree of priming after learning (score formalized in eq. \ref{['eqn:H']}) can be predicted from the keyword probability before learning.
• Figure 2: (a) For the 1320 Outlandish samples, the Pearson correlation between 8 basic measurements before learning, with the degree of priming they caused the LLM after learning ($\log \mathcal{S}_\text{prime}$). (b) expanded view of the measurement with the highest average correlation: keyword probability, with separate plots (red dots) for each of the 12 keywords (110 samples each: Section \ref{['sec:terminology']}). Each of the 12 plots displays keyword probability vs priming score $\mathcal{S}_\text{mem}$. Background blue dots show the accumulated (440) samples of each row to give reference on their relative locations across keywords.
• Figure 3: Relationship between keyword probability vs priming $\mathcal{S}_\text{prime}$ for PALM-2-xs undergoing spaced training, (a) for different spacings, and (b) for a particular spacing (1 outlandish sample presented once every $K=20$ iterations), plotted over number of presentations of Outlandish.
• Figure 4: Plot showing the change in $\log \mathcal{S}_\text{prime}$ vs the change in $\log \mathcal{S}_\text{mem}$ through the course of the first 5 gradient steps, across Outlandish samples, for PALM-2-xs, Llama-7b, and Gemma-2b models, showing different degrees of coupling between memorization vs priming across these different models.
• Figure 5: "Stepping stone" text augmentation strategy. (a-c) stepping stone text augmentation causes the keyword probability to drastically increase (c), while simultaneously - (a) causing the priming ($\mathcal{S}_\text{prime}$) to attenuate. Memorization ($\mathcal{S}_\text{mem}$) is intact (a). (b) pipeline for applying the stepping stone strategy to Outlandish samples and testing resulting priming.