Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models

Yungi Kim, Hyunsoo Ha, Seonghoon Yang, Sukyung Lee, Jihoo Kim, Chanjun Park

TL;DR

The Lightweight, Purpose-driven (LP) Data Pipeline is introduced, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation and enables the creation of purpose-driven datasets tailored to specific domains and languages.

Abstract

Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.

LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models

TL;DR

The Lightweight, Purpose-driven (LP) Data Pipeline is introduced, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation and enables the creation of purpose-driven datasets tailored to specific domains and languages.

Abstract

Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.

Paper Structure

This paper contains 26 sections, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Web Data Curation for LLMs
  4. Quality Filtering Techniques
  5. Lightweight, Purpose-driven (LP) Data Pipeline
  6. Pipeline Philosophy
  7. Fully CPU-Based Data Curation.
  8. Optimized Processing Order for Efficiency.
  9. Continuous Knowledge Updating.
  10. Purpose-Driven Datset Construction.
  11. Curation Pipeline
  12. Raw Text Extraction and URL Filtering
  13. Language Identification
  14. Line-Level Deduplication with Domain Grouping
  15. Heuristic Filtering
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Overview of the Lightweight, Purpose-driven (LP) Data Pipeline: Data extraction and cleansing flow from Common Crawl WARC dumps, illustrating the filtering processes and the sizes of data being filtered at each stage.
  • Figure 2: Overview of the training process for the quality filtering model and the domain classification model.
  • Figure 3: The token distribution of domain-specific datasets obtained from processing 10 CommonCrawl dumps for English and Korean using the Lightweight, Purpose-driven (LP) Data Pipeline.
  • Figure 4: System Architecture of the Lightweight, Purpose-driven (LP) Data Pipeline