Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

Zichun Yu, Chenyan Xiong

TL;DR

RePro tackles the data scarcity bottleneck in large language model pretraining by training a relatively small rephraser with reinforcement learning to produce high-quality, faithful rewrites of organic web data. It uses one quality reward (DataMan) and three faithfulness rewards (BERTScore, structure, length) to preserve semantics, structure, and length distributions while expanding usable training data. Empirical results on 22 downstream tasks show merit: recycled data yields 4.7–14.0% relative improvements over organic baselines and outperforms the ReWire method that uses a much larger rephraser, all while achieving 2–3× organic data efficiency and significantly higher data-processing efficiency. The work demonstrates that cost-efficient, faithfulness-aware data recycling can mitigate data-wall constraints in LLM pretraining and provides openly available code, rephraser models, and recycled data.

Abstract

High-quality pretraining data is the fossil fuel of large language models (LLMs), yet its reserves are running low for frontier models. In this paper, we introduce RePro, a novel web recycling method that trains a relatively small LM with reinforcement learning to generate effective and faithful rephrasings of pretraining data. Specifically, we design one quality reward and three faithfulness rewards, optimizing the LM rephraser to convert organic data into high-quality rephrasings while maintaining its core semantics and structure. In our experiment, we train a 4B rephraser to recycle 72B tokens sampled from DCLM-RefinedWeb. Pretraining results on 400M and 1.4B models demonstrate that RePro delivers 4.7%-14.0% relative accuracy gains over organic-only baseline on 22 downstream tasks. RePro also outperforms ReWire, the state-of-the-art web recycling method that prompts a 70B rephraser, as well as the organic baseline with a 4x larger data pool. Experiments with different amounts of recycled data highlight that RePro improves organic data efficiency by 2-3x. Individual and distributional analyses validate that RePro preserves more critical information and faithfully reflects the characteristics of organic data compared to prompting-based methods. Together, these results show that RePro provides an efficient and controllable path to effectively harness the fossil fuel of LLM pretraining. We open-source our code, rephraser, and recycled data at https://github.com/cxcscmu/RePro.

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

TL;DR

RePro tackles the data scarcity bottleneck in large language model pretraining by training a relatively small rephraser with reinforcement learning to produce high-quality, faithful rewrites of organic web data. It uses one quality reward (DataMan) and three faithfulness rewards (BERTScore, structure, length) to preserve semantics, structure, and length distributions while expanding usable training data. Empirical results on 22 downstream tasks show merit: recycled data yields 4.7–14.0% relative improvements over organic baselines and outperforms the ReWire method that uses a much larger rephraser, all while achieving 2–3× organic data efficiency and significantly higher data-processing efficiency. The work demonstrates that cost-efficient, faithfulness-aware data recycling can mitigate data-wall constraints in LLM pretraining and provides openly available code, rephraser models, and recycled data.

Abstract

High-quality pretraining data is the fossil fuel of large language models (LLMs), yet its reserves are running low for frontier models. In this paper, we introduce RePro, a novel web recycling method that trains a relatively small LM with reinforcement learning to generate effective and faithful rephrasings of pretraining data. Specifically, we design one quality reward and three faithfulness rewards, optimizing the LM rephraser to convert organic data into high-quality rephrasings while maintaining its core semantics and structure. In our experiment, we train a 4B rephraser to recycle 72B tokens sampled from DCLM-RefinedWeb. Pretraining results on 400M and 1.4B models demonstrate that RePro delivers 4.7%-14.0% relative accuracy gains over organic-only baseline on 22 downstream tasks. RePro also outperforms ReWire, the state-of-the-art web recycling method that prompts a 70B rephraser, as well as the organic baseline with a 4x larger data pool. Experiments with different amounts of recycled data highlight that RePro improves organic data efficiency by 2-3x. Individual and distributional analyses validate that RePro preserves more critical information and faithfully reflects the characteristics of organic data compared to prompting-based methods. Together, these results show that RePro provides an efficient and controllable path to effectively harness the fossil fuel of LLM pretraining. We open-source our code, rephraser, and recycled data at https://github.com/cxcscmu/RePro.

Paper Structure

This paper contains 27 sections, 6 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Recycling Pretraining Data
  5. Training a Faithful Rephraser with RL
  6. Experimental Setup
  7. Evaluation Results
  8. Main Results
  9. Overall Performance.
  10. Improving Organic Data Efficiency.
  11. Ablation Studies
  12. Effectiveness of Reward Functions
  13. Distributional Analyses
  14. Rephrasing Operations
  15. Conclusion
  16. ...and 12 more sections

Figures (9)

  • Figure 1: (a) Pretraining performance of 1.4B model and (b) ratio of key points in organic data that are supported / omitted / contradicted in the recycled version across three web recycling methods (rephraser size in parentheses).
  • Figure 2: Overview of RePro. We train our rephraser with quality and faithfulness rewards to optimize it to faithfully recycle the web.
  • Figure 3: Performance of RePro w.r.t. different unique token budgets.
  • Figure 4: Validation curves of (a) DataMan, (b) BERTScore, (c) structure, and (d) length rewards during our RL training with and without faithfulness rewards.
  • Figure 5: RL training with DCLM-fastText or data influence as quality rewards.
  • ...and 4 more figures