Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior

TL;DR

The paper tackles how pretraining data shape language-model behavior by proposing a formal, reusable interventional recipe that tests causal links through counterfactual retraining. It defines three stages—selecting evaluation items, identifying relevant documents, and intervening on those documents before retraining—and demonstrates the approach via two case studies on factual knowledge: term cooccurrence and information-retrieval–based queries. The findings show that manipulating training data can modulate knowledge learning, but simple interventions do not fully explain how facts are acquired, highlighting the need for more nuanced document selection and broader interventions. The work provides a practical framework and open-source guidance to study data-behavior relationships in LMs, enabling researchers to test new hypotheses about data effects.

Abstract

We present an experimental recipe for studying the relationship between training data and language model (LM) behavior. We outline steps for intervening on data batches -- i.e., ``rewriting history'' -- and then retraining model checkpoints over that data to test hypotheses relating data to behavior. Our recipe breaks down such an intervention into stages that include selecting evaluation items from a benchmark that measures model behavior, matching relevant documents to those items, and modifying those documents before retraining and measuring the effects. We demonstrate the utility of our recipe through case studies on factual knowledge acquisition in LMs, using both cooccurrence statistics and information retrieval methods to identify documents that might contribute to knowledge learning. Our results supplement past observational analyses that link cooccurrence to model behavior, while demonstrating that extant methods for identifying relevant training documents do not fully explain an LM's ability to correctly answer knowledge questions. Overall, we outline a recipe that researchers can follow to test further hypotheses about how training data affects model behavior. Our code is made publicly available to promote future work.

Figures (9)

• Figure 1: Diagram illustrating our proposed interventional analysis recipe, with stages including selecting items, matching documents to items, and intervening on matched documents.
• Figure 2: Results for replacing matching documents and re-training checkpoints to suppress learning of selected items from ParaRel. Accuracies are shown on the subset of target items. We include results of retraining on the original data batch (blue), training on a modified batch that replaces cooccurrences (orange), and training on a modified batch that (more aggressively) replaces occurrences (green). We also show results on target items (colored bars) and control items (crosses; i.e., all items other than the target group). For all results, we perform 5 repeated training runs and show the mean and one standard deviation across runs.
• Figure 3: Results for replacing matching documents and retraining to promote learning. Replacing documents not containing a cooccurrence with documents that do improves performance for all models and checkpoints.
• Figure 4: Toy example that shows how token IDs in a batch of tokens are altered when replacing a target document with a source document. Note how the token IDs to the right of the target document are shifted to the right to accommodate the longer source document.
• Figure 5: Percentage of documents labeled relevant to their matching item by GPT-5, for every dataset, prompt, and matching method.