Instruction Pre-Training: Language Models are Supervised Multitask Learners

TL;DR

Instruction Pre-Training augments raw corpora with instruction–response pairs generated by an open-source instruction synthesizer to enable supervised multitask pre-training. The approach yields data-efficient gains in general pre-training and robust improvements in domain-adaptive continual pre-training, including parity with substantially larger models in finance and biomedicine. Key contributions include the design of data collection, tuning, and inference for the synthesizer, extensive ablations, and analyses demonstrating high task diversity and practical usefulness of synthetic data. This work offers a scalable, cost-effective path to enhance generalization and domain adaptability of language models through supervised multitask signals embedded in pre-training.

Abstract

Unsupervised multitask pre-training has been the critical method behind the recent success of language models (LMs). However, supervised multitask learning still holds significant promise, as scaling it in the post-training stage trends towards better generalization. In this paper, we explore supervised multitask pre-training by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train LMs. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-Training. In pre-training from scratch, Instruction Pre-Training not only consistently enhances pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, Instruction Pre-Training enables Llama3-8B to be comparable to or even outperform Llama3-70B. Our model, code, and data are available at https://github.com/microsoft/LMOps.

Figures (7)

• Figure 1: Comparison between Instruction Pre-Training and Vanilla Pre-Training. Instead of directly pre-training on raw corpora, Instruction Pre-Training augments the corpora with instruction-response pairs generated by an instruction synthesizer, then pre-trains LMs on the augmented corpora. "Ins" and "Res" represent instruction and response, respectively.
• Figure 2: Tuning and inference framework of instruction synthesizer. During tuning, the instruction synthesizer learns to generate instruction-response pairs for a given raw text. The tuning data are curated to be highly diverse, enabling the synthesizer to generalize to unseen data. During inference, we use this tuned instruction synthesizer to generate instruction-response pairs for raw texts from pre-training corpora.
• Figure 3: For instruction synthesizer, a one-shot example consists of a raw text (T$_\text{N}$) and a set of instruction-response pairs (I$_\text{N}$$\bigoplus$R$_\text{N}$); data denoted without $'$ are for tuning the instruction synthesizer, and data with $'$ are for synthesizer inference and LM pre-training. During instruction synthesizer tuning, each sequence fed into the synthesizer concatenates multiple one-shot examples sampled from the same dataset. During inference, multi-round inference is conducted to synthesize instruction-response pairs with patterns similar to those of previous rounds. For LM pre-training, a few-shot example concatenates raw texts and synthesized pairs from multiple rounds.
• Figure 4: MMLU performance during instruction tuning of models pre-trained via Vanilla Pre-Training (Vanilla PT) and Instruction Pre-Training (Instruct PT).
• Figure 5: Helpfulness on LM generalization measured by LM performance with or without synthesized instruction-response pairs in the prompt.