How Does Code Pretraining Affect Language Model Task Performance?

TL;DR

The study causally probes how varying the proportion of code in pretraining data affects downstream performance, by constructing interleaved natural-language and code corpora under two regimes: competitive (fixed total data) and additive (fixed language data). Using decoder-only transformers, the authors evaluate on compositional generalization benchmarks (COGS, COGS-vf, English Passivization) and on BigBench tasks, revealing that more code improves structured-output generalization and arithmetic tasks but can harm linguistic and world-knowledge performance. Permutation tests show that code increases performance variance and boosts upper-quartile gains in aggregate, supporting nuanced guidance on data-source composition depending on target domains. The findings highlight domain-dependent trade-offs and suggest that optimal pretraining mixtures should align with intended downstream applications, while noting limitations related to scale, data separation, and task scope.

Abstract

Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language and code data. Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: additive, in which the total volume of data seen during pretraining is held constant; and competitive, in which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) a diverse collection of tasks included in the BigBench benchmark, and (b) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformations. We find that pretraining on higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increase code mixture can harm performance on other tasks, including on tasks that requires sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.

Figures (8)

• Figure 1: Code mixtures for the competitive and additive settings; for comparability between settings, we set the $0\%$ mixture in both settings to have 132 of natural-language data (so $N_\text{total} = 132\billion\tokens$ in the competitive setting, and $N_\text{lang} = 132\billion\tokens$ in the additive setting). Note that these sequences are shuffled during training, so models see code and language data at the same time.
• Figure 2: Full-sequence accuracy on the generalization set increases with code mixture on COGS and COGS-vf in both the competitive and additive settings. In the additive setting, code-mixture models outperform language-only baselines on the harder structural generalization cases. In all cases, validation accuracy is 100%.
• Figure 3: Pretraining code mixture has little impact on the well-formedness of generalization outputs in any setting.
• Figure 4: On multi-digit multiple choice arithmetic tasks, performance modestly increases with code mixture in the additive setting, while it increases then decreases in competitive. In both settings, the effect is more pronounced as the number of digits (rows) increases.
• Figure 5: On English Passivization, a compositional generalization benchmark where (unlike COGS) both the inputs and outputs are in natural language, increased code mixture results in lower full-sequence generalization accuracy in both settings. In the additive setting, code-mixture models underperform language-only baselines on the harder structural generalization cases.