Train Once, Answer All: Many Pretraining Experiments for the Cost of One

TL;DR

The results suggest that performing multiple pretraining experiments within a single training run can enable rigorous scientific experimentation with large models on a compute budget.

Abstract

Recent work has demonstrated that controlled pretraining experiments are a powerful tool for studying the relationship between training data and large language model (LLM) behavior. However, the computational cost of pretraining presents a significant constraint. To overcome this constraint, we propose a new approach where multiple experiments are conducted simultaneously during a single training run. We validate our approach by performing ten experiments while training on 210B tokens, with models of up to 2.7B parameters. Although models are trained only once, we can replicate the results of multiple previous works on data contamination, poisoning, and memorization. We also conduct novel investigations into knowledge acquisition, mathematical reasoning, and watermarking. For example, we dynamically update the training data until a model acquires a particular piece of knowledge. Remarkably, the influence of the experiments on the model's training dynamics and overall performance is minimal. However, interactions between experiments may act as a confounder in our approach. We propose continual pretraining dependence testing (CPDT), a novel technique to test for interactions with continual pretraining experiments, finding them to be negligible in our setup. Overall, our results suggest that performing multiple pretraining experiments within a single training run can enable rigorous scientific experimentation with large models on a compute budget.

Figures (33)

• Figure 1: We propose to conduct multiple independent pretraining experiments in a single training run.Top: Previous research performs one experiment per training run, then measures the outcome of this experiment. Bottom: In contrast, we propose to conduct multiple experiments simultaneously during a single training run, allowing us to measure the outcomes of multiple experiments while training only once.
• Figure 2: Results of the three novel experiments.(a): Algorithm \ref{['alg:control']} successfully maintains the value of the knowledge probe (blue) close to the control target (gray). (b): OLMo-2-1B-Exp exhibits a small degree of length-generalization to complex mathematical reasoning problems. (c): Gaussian Pretraining Watermarks are detectable over the course of training.
• Figure 3: Results of three replicated experiments.(a): Minor benchmark contamination is almost completely forgotten, consistent with bordt2025forgetting. (b): Rare tokens provide the most powerful canaries, replicating the findings of panda2025privacy. (c): The poisoned model allows for prompt extraction with the trigger string, corroborating zhang2025persistent.
• Figure 4: Comparing canary efficacy under 1-time canary inclusions. Point estimates are complemented by 95% bootstrap confidence intervals.
• Figure 5: The effect of model size on the results of three different experiments.(a): The effect of benchmark contamination increases with model size. (b): The prompt extraction attack remains successful at the 179M parameter scale. (c): Smaller models are less accurate at answering factual questions about the fictitious knowledge inserted as part of the knowledge acquisition experiment.

Theorems & Definitions (1)

• Definition 5.1: Experiment Independence