Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

Jiantao Qiu, Haijun Lv, Zhenjiang Jin, Rui Wang, Wenchang Ning, Jia Yu, ChaoBin Zhang, Zhenxiang Li, Pei Chu, Yuan Qu, Jin Shi, Lindong Lu, Runyu Peng, Zhiyuan Zeng, Huanze Tang, Zhikai Lei, Jiawei Hong, Keyu Chen, Zhaoye Fei, Ruiliang Xu, Wei Li, Zhongying Tu, Lin Dahua, Yu Qiao, Hang Yan, Conghui He

TL;DR

<3-5 sentence high-level summary> WanJuan-CC addresses the challenge of building large-scale language model training data with safety and quality constraints by proposing a rigorous, multi-stage pipeline for Common Crawl-derived English webtext. The approach combines text extraction, heuristic normalization, MinHash-LSH deduplication, and model-based safety and quality filtering to produce 2.22T tokens of safe data and 1.0T tokens of high-quality data, with 100B tokens open-sourced. Empirical evaluation using 1B and 3B parameter models shows WanJuan-CC offers improvements in validation perplexity and several downstream tasks over RefinedWeb, while achieving stronger safety metrics via Perspective API. The work provides a valuable, openly-available resource and a methodological blueprint for safe, high-quality webtext curation at scale.

Abstract

This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for language models, which require vast amounts of high-quality data. A comprehensive process was designed to handle Common Crawl data, including extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality filtering. From approximately 68 billion original English documents, we obtained 2.22T Tokens of safe data and selected 1.0T Tokens of high-quality data as part of WanJuan-CC. We have open-sourced 100B Tokens from this dataset. The paper also provides statistical information related to data quality, enabling users to select appropriate data according to their needs. To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb. Results show that WanJuan-CC performs better on validation datasets and downstream tasks.

WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

TL;DR

<3-5 sentence high-level summary> WanJuan-CC addresses the challenge of building large-scale language model training data with safety and quality constraints by proposing a rigorous, multi-stage pipeline for Common Crawl-derived English webtext. The approach combines text extraction, heuristic normalization, MinHash-LSH deduplication, and model-based safety and quality filtering to produce 2.22T tokens of safe data and 1.0T tokens of high-quality data, with 100B tokens open-sourced. Empirical evaluation using 1B and 3B parameter models shows WanJuan-CC offers improvements in validation perplexity and several downstream tasks over RefinedWeb, while achieving stronger safety metrics via Perspective API. The work provides a valuable, openly-available resource and a methodological blueprint for safe, high-quality webtext curation at scale.

Abstract

This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for language models, which require vast amounts of high-quality data. A comprehensive process was designed to handle Common Crawl data, including extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality filtering. From approximately 68 billion original English documents, we obtained 2.22T Tokens of safe data and selected 1.0T Tokens of high-quality data as part of WanJuan-CC. We have open-sourced 100B Tokens from this dataset. The paper also provides statistical information related to data quality, enabling users to select appropriate data according to their needs. To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb. Results show that WanJuan-CC performs better on validation datasets and downstream tasks.
Paper Structure (26 sections, 1 equation, 5 figures, 4 tables)

This paper contains 26 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Webtext Datasets
  4. Data Utility
  5. Data Safety
  6. Data Quality
  7. Method
  8. Processing Pipeline
  9. Data Extraction
  10. Heuristic Rule Filtering
  11. Deduplication with MinHash-LSH
  12. Safety Filtering
  13. Block Domain/Word List
  14. Toxicity Classification
  15. Pornography Classification
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Data Processing Pipeline. "Warc" represents the WARC format data from Common Crawl, "Eng" represents normal English data, "&*#" represents data with formatting errors, "Dup" represents duplicate data, "Porn" and "Toxic" represent pornographic and toxic data, respectively. "ADs" and "Bad" represent advertisements and low-quality data, respectively, while "Good" represents high-quality data. The red cross indicates data discarded at that stage.
  • Figure 2: Data Quality Evaluation Workflow
  • Figure 3: Documents Retention Rate and Removal Rate for Different Stages
  • Figure 4: Proportion of High-Quality Data from Different Common Crawl Dump Years
  • Figure 5: Percentage Statistics for Different Metrics on WanJuan-CC. To illustrate the main distribution area, the statistical range of some metrics has been truncated due to the presence of long-tail distributions.