Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation

Hanif Rahman

Abstract

Pashto is spoken by approximately 60--80 million people but has no published benchmarks for multilingual automatic speech recognition (ASR) on any shared public test set. This paper reports the first reproducible multi-model evaluation on public Pashto data, covering zero-shot ASR, script-level failure, and cross-domain evaluation of fine-tuned models. For zero-shot ASR, ten models (all seven Whisper sizes, MMS-1B, SeamlessM4T-v2-large, and OmniASR-CTC-300M) are evaluated on the FLEURS Pashto test set and a filtered Common Voice~24 subset; zero-shot Whisper WER ranges from 90% to 297%, with the medium model collapsing to 461% on Common Voice~24 consistent with decoder looping. SeamlessM4T achieves 39.7% WER on Common Voice~24 (the best zero-shot result reported to date, as of submission); MMS-1B achieves 43.8% on FLEURS. For script failure, a language-identification audit shows that no Whisper model produces Pashto-script output in more than 0.8% of utterances, while MMS-1B, SeamlessM4T, and OmniASR each exceed 93% Pashto-script fidelity; WER alone does not reveal this failure, since a model generating Arabic-script output on Pashto audio has not achieved ASR in any interpretable sense. For cross-domain evaluation, five fine-tuned Pashto ASR models are evaluated on both test sets: published WER figures of 14% degrade to 32.5--59% on out-of-distribution sets, while one augmented model achieves 35.1% on both sets with zero cross-domain degradation. Character-class error stratification confirms that Pashto-unique phonemes (the retroflex series and lateral fricatives) account for disproportionate error mass. All evaluations cover read speech only. Five structural impediments to cumulative progress are identified and five ordered research priorities are argued.

Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation

Abstract

Pashto is spoken by approximately 60--80 million people but has no published benchmarks for multilingual automatic speech recognition (ASR) on any shared public test set. This paper reports the first reproducible multi-model evaluation on public Pashto data, covering zero-shot ASR, script-level failure, and cross-domain evaluation of fine-tuned models. For zero-shot ASR, ten models (all seven Whisper sizes, MMS-1B, SeamlessM4T-v2-large, and OmniASR-CTC-300M) are evaluated on the FLEURS Pashto test set and a filtered Common Voice~24 subset; zero-shot Whisper WER ranges from 90% to 297%, with the medium model collapsing to 461% on Common Voice~24 consistent with decoder looping. SeamlessM4T achieves 39.7% WER on Common Voice~24 (the best zero-shot result reported to date, as of submission); MMS-1B achieves 43.8% on FLEURS. For script failure, a language-identification audit shows that no Whisper model produces Pashto-script output in more than 0.8% of utterances, while MMS-1B, SeamlessM4T, and OmniASR each exceed 93% Pashto-script fidelity; WER alone does not reveal this failure, since a model generating Arabic-script output on Pashto audio has not achieved ASR in any interpretable sense. For cross-domain evaluation, five fine-tuned Pashto ASR models are evaluated on both test sets: published WER figures of 14% degrade to 32.5--59% on out-of-distribution sets, while one augmented model achieves 35.1% on both sets with zero cross-domain degradation. Character-class error stratification confirms that Pashto-unique phonemes (the retroflex series and lateral fricatives) account for disproportionate error mass. All evaluations cover read speech only. Five structural impediments to cumulative progress are identified and five ordered research priorities are argued.

Paper Structure

This paper contains 31 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Phonological and Orthographic Background
  3. Pashto-Unique Phonemes
  4. Orthographic Challenges
  5. Related Work
  6. Experimental Setup
  7. Datasets
  8. Models
  9. Evaluation Protocol
  10. Zero-Shot ASR Benchmarks
  11. Whisper Across All Model Sizes
  12. WER on CV24 exceeds 100% for all Whisper models.
  13. The medium model collapses.
  14. Two distinct Whisper failure modes.
  15. Turbo versus large-v3.
  16. ...and 16 more sections

Figures (5)

  • Figure 1: Zero-shot WER for all ten models on FLEURS (left) and CV24_filtered (right), read speech only. Bars truncated at 320% (FLEURS) and 500% (CV24) for readability; Whisper-medium CV24 WER is 461%. Dashed line at WER = 100%.
  • Figure 2: Script output distribution per model on FLEURS (zero-shot only). Models ordered by Pashto% ascending.
  • Figure 3: Cross-domain generalisation: FLEURS WER ($x$-axis) vs. CV24 WER ($y$-axis). Points above the diagonal degrade on CV24 relative to FLEURS. WER capped at 130%. Circles: zero-shot; squares: fine-tuned.
  • Figure 4: WER on FLEURS and CV24_filtered for the four best zero-shot models and all five fine-tuned models. Dashed line separates zero-shot (left) from fine-tuned (right). The gap between FLEURS and CV24 bars shows cross-domain degradation; w2v-b2-aug shows no gap.
  • Figure 5: WER deviation from overall (35.4%) by character class for pashto-asr-v3 on FLEURS. Classes marked [Pashto-unique] are absent from Arabic, Dari, and Urdu.