Hubble: a Model Suite to Advance the Study of LLM Memorization

TL;DR

The paper presents Hubble, an open-source suite of Llama-3-based models (1B and 8B, trained on 100B or 500B tokens) with standard and perturbed variants to enable controlled memorization research across copyright, privacy, and test-set contamination domains. Through randomized perturbations and careful decontamination, Hubble enables precise measurement of memorization dynamics, revealing that increasing corpus size (dilution) and presenting sensitive data earlier in training mitigate memorization, while larger models are more prone to memorization at a given duplication level. Domain-specific analyses uncover nuanced memorization patterns, including PII leakage from synthetic biographies and indirect leakage in chats, and demonstrate Hubble’s utility for evaluating membership inference attacks and unlearning methods. The work provides a rigorous, transparent resource for benchmarking memorization, guiding mitigation strategies, and accelerating future research in understanding and safely managing memorization in large language models.

Abstract

We present Hubble, a suite of fully open-source large language models (LLMs) for the scientific study of LLM memorization. Hubble models come in standard and perturbed variants: standard models are pretrained on a large English corpus, and perturbed models are trained in the same way but with controlled insertion of text (e.g., book passages, biographies, and test sets) designed to emulate key memorization risks. Our core release includes 8 models -- standard and perturbed models with 1B or 8B parameters, pretrained on 100B or 500B tokens -- establishing that memorization risks are determined by the frequency of sensitive data relative to size of the training corpus (i.e., a password appearing once in a smaller corpus is memorized better than the same password in a larger corpus). Our release also includes 6 perturbed models with text inserted at different pretraining phases, showing that sensitive data without continued exposure can be forgotten. These findings suggest two best practices for addressing memorization risks: to dilute sensitive data by increasing the size of the training corpus, and to order sensitive data to appear earlier in training. Beyond these general empirical findings, Hubble enables a broad range of memorization research; for example, analyzing the biographies reveals how readily different types of private information are memorized. We also demonstrate that the randomized insertions in Hubble make it an ideal testbed for membership inference and machine unlearning, and invite the community to further explore, benchmark, and build upon our work.

Figures (22)

• Figure 1: Inserting a perturbation. First, we sample a training sequence from the standard training process to be perturbed. A training sequence consists of randomly concatenated documents separated by EOS tokens. To perturb it, we sample a gap (denoted in red) between the documents and splice the perturbation into a training sequence (between two existing documents). Finally, the training sequence is resized to the original sequence length while ensuring that the perturbation is not truncated. Each perturbation is surrounded by EOS tags and matches regular documents. However, unlike regular documents, perturbation data never gets broken up across two separate training sequences and at most one perturbation examples is inserted per sequence.
• Figure 2: Memorization of sensitive data can be diluted by training on larger corpora. We report the base evaluations on a subset of tasks for the core 8B models trained on 100B and 500B tokens. The core runs are described in §\ref{['sec:models']} and evaluations are described in §\ref{['sec:evaluations']}. For the same duplicate level, memorization is weaker for the model trained on 500B tokens compared to 100B. Figure \ref{['fig:model_scale-dclm_500b']} compares these trends against the 1B models, and larger models memorize at lower duplications. These experiments represent multiple interventions in one training run, and Figure \ref{['fig:interference-full']} plots these results for our interference models, which confirm minimal interference across domains.
• Figure 3: Sensitive data can be forgotten without continued exposure. We report the performance of the Timing runs (1B models trained on 100B tokens, described in §\ref{['sec:models']}) where perturbations are inserted in different phases of pretraining (tuples denote the range of pretraining where texts are inserted). For reference, the standard and perturbed 1B parameter models are also plotted.
• Figure 4: Unlearning performance on with Hubble 8B in copyright and privacy. Three key reference points are included in each subplot: the perturbed model (), representing performance before unlearning; the standard model (), which is trained without perturbations; and the desired model (), which achieves standard model's performance on the forget set while retaining the perturbed model's performance elsewhere. Improvement is indicated by the arrow (). See Appendix \ref{['sec:unlearn_full_results']} for the full results.
• Figure 5: Core results on Copyright Passages. The first row evaluates memorization with the length-normalized log-likelihood of the models on the passages. The lower two rows measure the accuracy of verbatim generation, where the models are prompted to generate a 100-token continuation given a 50-token prefix.