How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP

TL;DR

This work interrogates Wikipedia as a data source for multilingual NLP, showing that non-English Wikipedias often contain noise such as foreign scripts and bot-generated content. It introduces a two-stage filtering pipeline—primary filtering (script-based language filtering and deduplication) followed by corpus-driven heuristic filters—and demonstrates that pruning bad data can preserve or improve downstream performance, especially for low-resource languages. Through monolingual, language-adaptation, and multilingual pretraining experiments, the paper reveals that data quality effects are language- and task-specific, with gains most pronounced in resource-constrained settings. The findings advocate for language- and domain-specific data quality definitions and provide actionable guidelines and code to help practitioners use Wikipedia more efficiently for pretraining and evaluation in multilingual NLP.

Abstract

Wikipedia's perceived high quality and broad language coverage have established it as a fundamental resource in multilingual NLP. In the context of low-resource languages, however, these quality assumptions are increasingly being scrutinised. This paper critically examines the data quality of Wikipedia in a non-English setting by subjecting it to various quality filtering techniques, revealing widespread issues such as a high percentage of one-line articles and duplicate articles. We evaluate the downstream impact of quality filtering on Wikipedia and find that data quality pruning is an effective means for resource-efficient training without hurting performance, especially for low-resource languages. Moreover, we advocate for a shift in perspective from seeking a general definition of data quality towards a more language- and task-specific one. Ultimately, we aim for this study to serve as a guide to using Wikipedia for pretraining in a multilingual setting.

Figures (10)

• Figure 1: Proportion of articles and characters filtered from non-English Wikipedias with primary filtering.
• Figure 2: Division of Wikipedia based on a clustering on fraction of documents removed and fraction of characters removed from primary filtering. The blue cluster corresponds to Tier 1, orange to Tier 2, green to Tier 3 and red to Tier 4. Each point is weighed by the article count of the unfiltered Wikipedia.
• Figure 3: A confusion matrix comparing the joshi-etal-2020-state and the Wikipedia quality tiers.
• Figure 4: Percentage of total docs removed by MinHash and ratio of bot articles to total articles per Wikipedia.
• Figure 5: Distribution of select Wikipedias from Tier 1 (blue), Tier 2 (orange), Tier 3 (green) and Tier 4 (red) on each class of heuristic metrics.