Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development

Hanif Rahman

Abstract

We present PashtoCorp, a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains severely underrepresented in NLP. The corpus is assembled from 39 sources spanning seven HuggingFace datasets and 32 purpose-built web scrapers, processed through a reproducible pipeline with Arabic-script tokenization, SHA-256 deduplication, and quality filtering. At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus. Continued MLM pretraining of XLM-R-base on PashtoCorp reduces held-out perplexity by 25.1% (8.08->6.06). On WikiANN Pashto NER, the pretrained model improves entity F1 by 10% relative (19.0%->21.0%) and reduces training variance nearly 7x; the largest gain appears at 50 training sentences (+27%), with PashtoCorp covering 97.9% of WikiANN entity vocabulary. On Belebele Pashto reading comprehension, Gemma-3n achieves 64.6% accuracy, the first published LLM baseline for Pashto on this benchmark. A leave-one-out source ablation shows that Wikipedia (0.7% of documents) is the most critical source for NER: removing it alone reduces entity F1 by 47%. Corpus data, trained model, and code are available at https://huggingface.co/datasets/ihanif/pashto-corpus, https://huggingface.co/ihanif/xlmr-pashto, and https://github.com/ihanif/pashto-corpus.

PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development

Abstract

We present PashtoCorp, a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains severely underrepresented in NLP. The corpus is assembled from 39 sources spanning seven HuggingFace datasets and 32 purpose-built web scrapers, processed through a reproducible pipeline with Arabic-script tokenization, SHA-256 deduplication, and quality filtering. At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus. Continued MLM pretraining of XLM-R-base on PashtoCorp reduces held-out perplexity by 25.1% (8.08->6.06). On WikiANN Pashto NER, the pretrained model improves entity F1 by 10% relative (19.0%->21.0%) and reduces training variance nearly 7x; the largest gain appears at 50 training sentences (+27%), with PashtoCorp covering 97.9% of WikiANN entity vocabulary. On Belebele Pashto reading comprehension, Gemma-3n achieves 64.6% accuracy, the first published LLM baseline for Pashto on this benchmark. A leave-one-out source ablation shows that Wikipedia (0.7% of documents) is the most critical source for NER: removing it alone reduces entity F1 by 47%. Corpus data, trained model, and code are available at https://huggingface.co/datasets/ihanif/pashto-corpus, https://huggingface.co/ihanif/xlmr-pashto, and https://github.com/ihanif/pashto-corpus.
Paper Structure (36 sections, 4 figures, 12 tables)

This paper contains 36 sections, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Collection
  4. HuggingFace Datasets
  5. Web Scraping
  6. Processing Pipeline
  7. Language Identification
  8. Deduplication
  9. Quality Filtering
  10. Corpus Analysis
  11. Scale and Comparison
  12. Domain Distribution
  13. Vocabulary Growth
  14. Source Contribution Ablation
  15. Vocabulary ablation.
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Document distribution by domain category ($N=2{,}810{,}913$).
  • Figure 2: Cumulative unique vocabulary (millions) as sources are incorporated in descending document-count order. Despite its small size (16.9K documents, position 17), FinePDFs contributes the largest marginal vocabulary of any single source: 2.22M unique types.
  • Figure 3: Log-log frequency vs. rank for the top $10^5$ word types. The Zipf fit ($\alpha = 1.624$, $R^2 = 0.985$) holds closely.
  • Figure 4: WikiANN NER entity F1 vs. training set size (5 seeds, $\pm$1 std shaded). PashtoCorp pretraining peaks at $n=50$ (+27% relative F1).