Procedural Pretraining: Warming Up Language Models with Abstract Data

TL;DR

Procedural pretraining introduces a warm-up phase using abstract procedural data to ease the entanglement of knowledge and reasoning in large language models. The approach uses data generated from simple algorithms and formal languages to teach elementary algorithmic skills before standard semantic pretraining, and it yields improvements in algorithmic tasks and downstream semantic domains. The study shows that pretrained information localizes differently across layers (attention for structured data, MLP for language) and that combining procedural data types provides additive gains, with scalability demonstrated up to $1.3\mathrm{B}$ parameters. Together, these findings point to a data-efficient path toward disentangling knowledge acquisition from reasoning in LLMs and suggest practical routes for modular transfer and accelerated pretraining.

Abstract

Pretraining directly on web-scale corpora is the de facto paradigm for building language models. We study an alternative setting where the model is initially exposed to abstract structured data, as a means to ease the subsequent acquisition of rich semantic knowledge, much like humans learn simple logic and mathematics before higher reasoning. We specifically focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, on context recall (Needle-in-a-haystack), the accuracy jumps from 10 to 98% when pretraining on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1% procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics (C4, CodeParrot, and DeepMind-Math datasets). Notably, this procedural pretraining enables the models to reach the same loss value with only 55, 67, 86% of the original data. Third, we explore the mechanisms behind and find that procedural pretraining instils non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Our results show that procedural pretraining is a simple, lightweight means to improving performance and accelerating language model pretraining, ultimately suggesting the promise of disentangling knowledge acquisition from reasoning in LLMs.

Figures (22)

• Figure 1: (Left) We pretrain language models on procedural data before exposing them to standard datasets of language, code, or mathematics. The procedural data is generated with simple algorithms and aims to teach elementary skills to aid the acquisition of semantic knowledge. (Right) This lightweight initial step speeds up standard pretraining and improves performance on diverse domains, with different pretrained layers (MLP vs. attention) contributing differently to each domain.
• Figure 2: Different types of procedural pretraining can significantly improve over standard training (dashed line) across various algorithmic tasks. If we remove the structure within the procedural data by shuffling the sequences (Best model shuffled), the performance falls to the baseline. Reported values are the means over 10 seeds (full results with variance in Appendix \ref{['app:more_results-algorthmic-reasoning']}).
• Figure 3: Selective transfer of MLP or attention layers can improve over full-model transfer, showing that procedural pretraining creates 'modular' structure localised in the selected model components. Reported values are means across 10 seeds (full results with variance in Appendix \ref{['app:more_results-algorthmic-reasoning']}).
• Figure 4: The benefits of procedural pretraining transfer to semantic domains. Perplexity (lower is better) on natural language (left) and pure code (right). A little of procedural data is very effective: compare the number of procedural tokens ($T_1$) in these plots with the amount of tokens from the target datasets ($T_2$) being 15M for WikiText and 105M for JavaCorpus.
• Figure 5: Procedural pretraining is complementary to standard data & highly data-efficient. Each column corresponds to a different semantic dataset. (Top) Training curves with different types of procedural data (Union, Sort, Set). (Middle) Additive setting: a small amount of procedural data is sufficient to outperform standard pretraining. (Bottom) Substitutive setting: we plot curves whose points $(x,y)$ achieve equivalent performance with $x$ procedural tokens and $y$ standard tokens. We can drastically reduce the total amount of data when using a small fraction of procedural data. Full-model transfer (see Section \ref{['sec:experimental_setup']}) is used for procedural pretraining.