GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
Amir Hossein Kargaran, François Yvon, Hinrich Schütze
TL;DR
GlotCC tackles the scarcity of large, clean web corpora for minority languages by delivering an open, document-level corpus built from CommonCrawl via an open Ungoliant-based pipeline and GlotLID v3.0 labeling. It significantly expands language coverage with over 2000 LID labels, integrates robust noise handling (zxx) and unseen-language filtering (und), and adds content-class and PII-removal steps, complemented by self-audit and cross-benchmark evaluation. The resulting GlotCC v1.0 covers 1275 language-script labels and demonstrates strong in-language quality, while openly acknowledging limitations in certain scripts and content domains. By releasing complete tooling and metadata under open licenses, the work lowers barriers for minority-language NLP research and enables reproducible, scalable data curation for language technologies.
Abstract
The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models. Most available corpora have sufficient data only for languages with large dominant communities. However, there is no corpus available that (i) covers a wide range of minority languages; (ii) is generated by an open-source reproducible pipeline; and (iii) is rigorously cleaned from noise, making it trustworthy to use. We present GlotCC, a clean, document-level, 2TB general domain corpus derived from CommonCrawl, covering more than 1000 languages. We make GlotCC and the system used to generate it - including the pipeline, language identification model, and filters - available to the research community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1, Pipeline v. 3.0 https://github.com/cisnlp/GlotCC.