Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks

TL;DR

The work reveals a pronounced positional bias in LLM memorization: verbatim recall is strongest for prefixes at the start of the context window and rapidly declines as the prefix is offset deeper in the window. By performing controlled pretraining on Gutenberg and web data across 1B, 3B, and 8B models, the authors demonstrate a robust offset effect, link memorization breakdown to text degeneration, and propose mitigation by shifting sensitive content deeper into the context window (as in Swapped Gutenberg). They further show that smaller batch sizes and targeted pretraining strategies like Goldfish Loss can suppress memorization while preserving or improving downstream performance. These findings establish offset as a critical, previously underexplored axis for memorization risk assessment and mitigation with practical implications for safer, more compliant LLM deployment.

Abstract

Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, we pretrain language models (1B/3B/8B) from scratch on 83B tokens, mixing web-scale data with public domain books used to simulate copyrighted content at controlled frequencies at lengths at least ten times longer than prior work. We thereby identified the offset effect, a phenomenon characterized by two key findings: (1) verbatim memorization is most strongly triggered by short prefixes drawn from the beginning of the context window, with memorization decreasing counterintuitively as prefix length increases; and (2) a sharp decline in verbatim recall when prefix begins offset from the initial tokens of the context window. We attribute this to positional fragility: models rely disproportionately on the earliest tokens in their context window as retrieval anchors, making them sensitive to even slight shifts. We further observe that when the model fails to retrieve memorized content, it often produces degenerated text. Leveraging these findings, we show that shifting sensitive data deeper into the context window suppresses both extractable memorization and degeneration. Our results suggest that positional offset is a critical and previously overlooked axis for evaluating memorization risks, since prior work implicitly assumed uniformity by probing only from the beginning of training sequences.

Figures (12)

• Figure 1: Positional fragility in LLMs measured with two complementary metrics: (a) Rouge-L scores quantify verbatim memorization, demonstrating a sharp decline as offset increases; (b) MAUVE scores capture overall language coherence, revealing text degeneration with increasing offset. The dense model results are derived from a 1B model trained exclusively on 10K Gutenberg sequences seen 80 times each, while sparse model results are derived specifically from the frequency-128 bucket of our FM-Probe methodology. All experiments used 500-token prefixes and evaluated on 500-token suffixes. Verbatim memorization and language coherence both degrade sharply as prefix offset increases, except for language coherence in 1B Dense case, revealing positional fragility.
• Figure 2: Illustration of prefix-suffix extraction with varying offsets from the document start in Sparse Gutenberg, and from the retained part in Swapped Gutenberg. Not to scale.
• Figure 3: Impact of prefix length on positional fragility across LLaMA models (1B, 3B, 8B), measured by Rouge-L scores on frequency-128 bucket with 500-token suffixes under Sparse Gutenberg setup. With zero offset (blue), smaller models show memorization degradation as prefix length increases, while the 8B model maintains high scores. At non-zero offsets (red, green), all models require longer prefixes to trigger substantial memorization, with the 3B model showing slightly higher memorization than the 8B model at offsets 64 and 128, indicating larger models do not worsen positional fragility.
• Figure 4: Effect of isolating the BOD token on the memorization capability of the 1B model measured by Rouge-L scores, computed over 500-token suffixes conditioned on varying prefix lengths (50–5000 tokens). Subplots from left to right correspond to context window offsets of 0, 50, and 100, respectively.
• Figure 5: Impact of exposure frequency on text quality metrics for Llama models of different sizes, measured with two complementary metrics: (a) TTR scores quantify lexical diversity, showing improvement as training frequency increases; the ground truth TTR lies in the range 0.535–0.541;(b) MAUVE scores capture overall language quality and coherence, reaching near-perfect scores at high frequencies. All experiments used 500-token prefixes and were evaluated on 500-token suffixes with a zero offset.