Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages

Jia Yu, Fei Yuan, Rui Min, Jing Yu, Pei Chu, Jiayang Li, Wei Li, Ruijie Zhang, Zhenxiang Li, Zhifei Ren, Dong Zheng, Wenjian Zhang, Yan Teng, Lingyu Meng, ZhenJiang Jin, Jiantao Qiu, ShaSha Wang, Zhongying Tu, Dahua Lin, Yu Wang, Yu Qiao, Yanfeng Wang, Conghui He

TL;DR

The paper tackles the lack of high-quality multilingual data for low-resource languages by proposing WanJuanSiLu, a large open-source multilingual text corpus built through a systematic data processing pipeline that emphasizes data cleaning, deduplication, quality screening, safety filtering, and thematic classification. The authors develop language-aware rules and a two-tier classification schema (8 domains, 34 subcategories) and validate the dataset via manual and automated quality assessments, comparing against public baselines. They demonstrate strong data quality, safety, and cross-language consistency, and show improvements in language-model pretraining performance when trained on the filtered data. This work provides a practical resource and a scalable framework for constructing high-quality multilingual datasets to advance low-resource language AI.

Abstract

This paper introduces the open-source dataset WanJuanSiLu, designed to provide high-quality training corpora for low-resource languages, thereby advancing the research and development of multilingual models. To achieve this, we have developed a systematic data processing framework tailored for low-resource languages. This framework encompasses key stages such as data extraction, corpus cleaning, content deduplication, security filtering, quality evaluation, and theme classification. Through the implementation of this framework, we have significantly improved both the quality and security of the dataset, while maintaining its linguistic diversity. As of now, data for all five languages have been fully open-sourced. The dataset can be accessed at https://opendatalab.com/applyMultilingualCorpus, and GitHub repository is available at https://github.com/opendatalab/WanJuan3.0

WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages

TL;DR

The paper tackles the lack of high-quality multilingual data for low-resource languages by proposing WanJuanSiLu, a large open-source multilingual text corpus built through a systematic data processing pipeline that emphasizes data cleaning, deduplication, quality screening, safety filtering, and thematic classification. The authors develop language-aware rules and a two-tier classification schema (8 domains, 34 subcategories) and validate the dataset via manual and automated quality assessments, comparing against public baselines. They demonstrate strong data quality, safety, and cross-language consistency, and show improvements in language-model pretraining performance when trained on the filtered data. This work provides a practical resource and a scalable framework for constructing high-quality multilingual datasets to advance low-resource language AI.

Abstract

This paper introduces the open-source dataset WanJuanSiLu, designed to provide high-quality training corpora for low-resource languages, thereby advancing the research and development of multilingual models. To achieve this, we have developed a systematic data processing framework tailored for low-resource languages. This framework encompasses key stages such as data extraction, corpus cleaning, content deduplication, security filtering, quality evaluation, and theme classification. Through the implementation of this framework, we have significantly improved both the quality and security of the dataset, while maintaining its linguistic diversity. As of now, data for all five languages have been fully open-sourced. The dataset can be accessed at https://opendatalab.com/applyMultilingualCorpus, and GitHub repository is available at https://github.com/opendatalab/WanJuan3.0
Paper Structure (12 sections, 5 figures, 7 tables)

This paper contains 12 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Dataset Statistics
  3. Method
  4. Data Clean
  5. Quality Evaluation
  6. Safety Filtering
  7. Theme Classification
  8. Result
  9. Analysis of Data Retention
  10. A Comprehensive Evaluation of Data Quality
  11. Quality Testing and Validation
  12. Conclusion

Figures (5)

  • Figure 1: Distribution of Themes Across Level 1 and Level 2 Labels in the Corpus
  • Figure 2: Data Processing Pipeline
  • Figure 3: Examples of Quality Issues
  • Figure 4: Diagram of Perplexity Values and Quality Scores
  • Figure 5: Retention and Removal Rate for Different Stages