Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data

Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, Xu Han, Zhiyuan Liu

TL;DR

This work tackles the quality and efficiency bottlenecks of model-driven data filtering for LLM pretraining by introducing an efficient verification strategy that leverages a near-final LLM with a two-stage annealing process. It pairs this with a robust seed-data framework and a FastText-based classifier to build Ultra-FineWeb from FineWeb and Chinese FineWeb, achieving substantial improvements on English and Chinese benchmarks while significantly reducing computational costs. The paper provides detailed methodology, extensive experiments, and multiple ablations demonstrating the value of multi-source seeds, intersection-based positive samples, and loss-estimation-guided validation. The result is a scalable, cost-effective pipeline for generating higher-quality pretraining data with practical implications for large-scale LLM training.

Abstract

Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become a primary approach for acquiring high-quality data. However, it still faces two main challenges: (1) the lack of an efficient data verification strategy makes it difficult to provide timely feedback on data quality; and (2) the selection of seed data for training classifiers lacks clear criteria and relies heavily on human expertise, introducing a degree of subjectivity. To address the first challenge, we introduce an efficient verification strategy that enables rapid evaluation of the impact of data on LLM training with minimal computational cost. To tackle the second challenge, we build upon the assumption that high-quality seed data is beneficial for LLM training, and by integrating the proposed verification strategy, we optimize the selection of positive and negative samples and propose an efficient data filtering pipeline. This pipeline not only improves filtering efficiency, classifier quality, and robustness, but also significantly reduces experimental and inference costs. In addition, to efficiently filter high-quality data, we employ a lightweight classifier based on fastText, and successfully apply the filtering pipeline to two widely-used pre-training corpora, FineWeb and Chinese FineWeb datasets, resulting in the creation of the higher-quality Ultra-FineWeb dataset. Ultra-FineWeb contains approximately 1 trillion English tokens and 120 billion Chinese tokens. Empirical results demonstrate that the LLMs trained on Ultra-FineWeb exhibit significant performance improvements across multiple benchmark tasks, validating the effectiveness of our pipeline in enhancing both data quality and training efficiency.

Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data

TL;DR

This work tackles the quality and efficiency bottlenecks of model-driven data filtering for LLM pretraining by introducing an efficient verification strategy that leverages a near-final LLM with a two-stage annealing process. It pairs this with a robust seed-data framework and a FastText-based classifier to build Ultra-FineWeb from FineWeb and Chinese FineWeb, achieving substantial improvements on English and Chinese benchmarks while significantly reducing computational costs. The paper provides detailed methodology, extensive experiments, and multiple ablations demonstrating the value of multi-source seeds, intersection-based positive samples, and loss-estimation-guided validation. The result is a scalable, cost-effective pipeline for generating higher-quality pretraining data with practical implications for large-scale LLM training.

Abstract

Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become a primary approach for acquiring high-quality data. However, it still faces two main challenges: (1) the lack of an efficient data verification strategy makes it difficult to provide timely feedback on data quality; and (2) the selection of seed data for training classifiers lacks clear criteria and relies heavily on human expertise, introducing a degree of subjectivity. To address the first challenge, we introduce an efficient verification strategy that enables rapid evaluation of the impact of data on LLM training with minimal computational cost. To tackle the second challenge, we build upon the assumption that high-quality seed data is beneficial for LLM training, and by integrating the proposed verification strategy, we optimize the selection of positive and negative samples and propose an efficient data filtering pipeline. This pipeline not only improves filtering efficiency, classifier quality, and robustness, but also significantly reduces experimental and inference costs. In addition, to efficiently filter high-quality data, we employ a lightweight classifier based on fastText, and successfully apply the filtering pipeline to two widely-used pre-training corpora, FineWeb and Chinese FineWeb datasets, resulting in the creation of the higher-quality Ultra-FineWeb dataset. Ultra-FineWeb contains approximately 1 trillion English tokens and 120 billion Chinese tokens. Empirical results demonstrate that the LLMs trained on Ultra-FineWeb exhibit significant performance improvements across multiple benchmark tasks, validating the effectiveness of our pipeline in enhancing both data quality and training efficiency.
Paper Structure (16 sections, 2 equations, 7 figures, 14 tables)

This paper contains 16 sections, 2 equations, 7 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overall Workflow
  4. Efficient Verification Strategy
  5. Classifier Training Seeds
  6. Classifier Training Recipes
  7. FastText-based Quality Filtering
  8. Experiments
  9. Experimental Setting
  10. Overall Results
  11. Ablation Study and Analysis
  12. Related Work
  13. Conclusion
  14. Limitations and Future Directions
  15. Implementation Details for Efficient Verification
  16. ...and 1 more sections

Figures (7)

  • Figure 1: Comparison of High-Quality Data Filtering Pipelines. Traditional model-based data filtering methods (a) and (b) rely on human expertise for seed data selection and lack data quality verification.
  • Figure 2: Average scores at each checkpoint for different individual datasets.
  • Figure 3: Average scores at each checkpoint for different mixed datasets.
  • Figure 4: Comparison of token length distributions across different datasets.
  • Figure 5: Loss-performance curve: Showing the estimated performance of an 8B model trained on 8T tokens using baseline and replacing high-quality data with Ultra-FineWeb.
  • ...and 2 more figures