Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Ajay Patel, Colin Raffel, Chris Callison-Burch

TL;DR

FineInstructions rethinks pre-training by replacing raw documents with billions of synthetic instruction–answer pairs generated from 18M templates instantiated on real docs. The pipeline comprises template generation, document–template matching with Gaussian-pooling-enhanced embeddings, instantiation of templates into grounded answers, and rigorous judging to filter high-quality data. Empirical results show pre-training on FineInstructions outperforms standard pre-training and other synthetic approaches across several benchmarks (MixEval, MT-Bench-101, AlpacaEval), with improvements that persist across model scales and curricula. This approach aligns pre-training data with downstream user usage, enabling more efficient knowledge absorption and instruction-following behavior at scale, and provides a foundation for future scalable, domain-diverse synthetic data generation.

Abstract

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

TL;DR

FineInstructions rethinks pre-training by replacing raw documents with billions of synthetic instruction–answer pairs generated from 18M templates instantiated on real docs. The pipeline comprises template generation, document–template matching with Gaussian-pooling-enhanced embeddings, instantiation of templates into grounded answers, and rigorous judging to filter high-quality data. Empirical results show pre-training on FineInstructions outperforms standard pre-training and other synthetic approaches across several benchmarks (MixEval, MT-Bench-101, AlpacaEval), with improvements that persist across model scales and curricula. This approach aligns pre-training data with downstream user usage, enabling more efficient knowledge absorption and instruction-following behavior at scale, and provides a foundation for future scalable, domain-diverse synthetic data generation.

Abstract

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .
Paper Structure (30 sections, 3 figures, 4 tables)

This paper contains 30 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. FineInstructions
  4. Generating Instruction Templates
  5. Matching Documents to Instruction Templates
  6. Gaussian Pooling for Document Coverage
  7. Retrieval
  8. Generating Instructions and Answers
  9. Judging and Filtering Instructions and Answers
  10. Experimental Setup
  11. Baselines
  12. Instruction Pre-Training (IPT)
  13. Nemotron-CC
  14. Pre-Training
  15. Benchmarks
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Efficiency from pre-training on FineInstructions data.
  • Figure 2: The FineInstructions pipeline for efficiently generating diverse, pre-training scale, synthetic instruction-answer pairs.
  • Figure 3: A visualization of the task diversity in FineInstructions. We also preview domain-specific diversity, comparing with a subset that was instantiated from medicine-related templates. Charts can be read from the inner ring to the outer ring for each "slice".