Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Integrated Data Processing Framework for Pretraining Foundation Models

Yiding Sun, Feng Wang, Yutao Zhu, Wayne Xin Zhao, Jiaxin Mao

TL;DR

The paper tackles the difficulty of obtaining high-quality, diverse pretraining data for foundation models by introducing an integrated data processing framework. It combines a Processing Module with multi-granularity operators and an Analyzing Module for probing and evaluation, augmented by a Retriever component to mitigate hallucinations through retrieval augmentation. Automated ChatGPT-based assessments and end-to-end GPT-2 pretraining on refined data show substantial quality improvements and faster initial learning, though the benefits must be balanced against data quantity for sustained gains. The framework offers a practical, flexible tool for dataset curation with code and demonstrations available on GitHub, aiming to streamline and standardize pretraining data pipelines.

Abstract

The ability of the foundation models heavily relies on large-scale, diverse, and high-quality pretraining data. In order to improve data quality, researchers and practitioners often have to manually curate datasets from difference sources and develop dedicated data cleansing pipeline for each data repository. Lacking a unified data processing framework, this process is repetitive and cumbersome. To mitigate this issue, we propose a data processing framework that integrates a Processing Module which consists of a series of operators at different granularity levels, and an Analyzing Module which supports probing and evaluation of the refined data. The proposed framework is easy to use and highly flexible. In this demo paper, we first introduce how to use this framework with some example use cases and then demonstrate its effectiveness in improving the data quality with an automated evaluation with ChatGPT and an end-to-end evaluation in pretraining the GPT-2 model. The code and demonstration videos are accessible on GitHub.

An Integrated Data Processing Framework for Pretraining Foundation Models

TL;DR

The paper tackles the difficulty of obtaining high-quality, diverse pretraining data for foundation models by introducing an integrated data processing framework. It combines a Processing Module with multi-granularity operators and an Analyzing Module for probing and evaluation, augmented by a Retriever component to mitigate hallucinations through retrieval augmentation. Automated ChatGPT-based assessments and end-to-end GPT-2 pretraining on refined data show substantial quality improvements and faster initial learning, though the benefits must be balanced against data quantity for sustained gains. The framework offers a practical, flexible tool for dataset curation with code and demonstrations available on GitHub, aiming to streamline and standardize pretraining data pipelines.

Abstract

The ability of the foundation models heavily relies on large-scale, diverse, and high-quality pretraining data. In order to improve data quality, researchers and practitioners often have to manually curate datasets from difference sources and develop dedicated data cleansing pipeline for each data repository. Lacking a unified data processing framework, this process is repetitive and cumbersome. To mitigate this issue, we propose a data processing framework that integrates a Processing Module which consists of a series of operators at different granularity levels, and an Analyzing Module which supports probing and evaluation of the refined data. The proposed framework is easy to use and highly flexible. In this demo paper, we first introduce how to use this framework with some example use cases and then demonstrate its effectiveness in improving the data quality with an automated evaluation with ChatGPT and an end-to-end evaluation in pretraining the GPT-2 model. The code and demonstration videos are accessible on GitHub.
Paper Structure (12 sections, 2 figures, 2 tables)

This paper contains 12 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. USE CASES
  3. Integrated Data Processing Framework
  4. Retriever Augmentation
  5. Implementation
  6. Processing Module
  7. Analyzing Module
  8. Experiments
  9. Automated Evaluation
  10. End-to-end Evaluation
  11. Related Works
  12. Conclusion

Figures (2)

  • Figure 1: Overview of our data processing framework.
  • Figure 2: Performance as a function of training steps. The left illustrates the decreasing loss on the validation set, while the right indicates trends of PPL on LAMBADA and WikiText103.