The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR

TL;DR

The paper addresses how much audio is needed to observe a multilingual ASR model's sub-token inventory and whether training-data disparities shape inference-time token usage. Using Whisper and Common Voice 17 across 49 languages, it logs decoder candidate sub-tokens and models discovery trajectories with an exponential saturation to define acoustic saturation time $T_{90}$. Key findings show that total sub-token discovery is largely independent of pre-training hours, saturation occurs near two hours of audio, and rank-frequency patterns align with Zipf–Mandelbrot, with script-specific differences. The results imply that sub-token utilization is driven more by orthographic and typological structure than data scale, informing equitable corpus design and cross-lingual evaluation, while offering AST as a practical criterion for corpus sufficiency in multilingual ASR.

Abstract

How much audio is needed to fully observe a multilingual ASR model's learned sub-token inventory across languages, and does data disparity in multilingual pre-training affect how these tokens are utilized during inference? We address this question by analyzing Whisper's decoding behavior during inference across 49 languages. By logging decoding candidate sub-tokens and tracking their cumulative discovery over time, we study the utilization pattern of the model's sub-token space. Results show that the total number of discovered tokens remains largely independent of a language's pre-training hours, indicating that data disparity does not strongly influence lexical diversity in the model's hypothesis space. Sub-token discovery rates follow a consistent exponential saturation pattern across languages, suggesting a stable time window after which additional audio yields minimal new sub-token activation. We refer to this convergence threshold as acoustic saturation time (AST). Further analyses of rank-frequency distributions reveal Zipf-like patterns better modeled by a Zipf-Mandelbrot law, and mean sub-token length shows a positive correlation with resource level. Additionally, those metrics show more favorable patterns for languages in the Latin script than those in scripts such as Cyrillic, CJK, and Semitic. Together, our study suggests that sub-token utilization during multilingual ASR inference is constrained more by the statistical, typological, and orthographic structure of the speech than by training data scale, providing an empirical basis for more equitable corpus construction and cross-lingual evaluation.

Figures (5)

• Figure 1: Character Error Rate (CER) at 10 minutes for all 65 languages, sorted by CER. The red dashed line denotes the 30% inclusion threshold; only languages below the line (n=49) are included.
• Figure 2: Sub-token discovery after 120 minutes across 49 languages. Each point represents one language, with color indicating training data hours and dashed line showing the logarithmic trend. Training data hours show weak, non-significant correlation with sub-token count.
• Figure 3: Token saturation curves across 46 languages. Thin lines show individual languages; thick lines show script median. Diamonds mark 90% saturation points ($T_{90}$). Red vertical dashed line indicates the collection window at 120 minutes.
• Figure 4: Rank-frequency distributions on a log-log scale, grouped by writing system. Thin lines show individual languages; thick lines show script-level medians with IQR ribbons. The dashed reference line indicates canonical Zipf behavior ($\alpha = 1$).
• Figure 5: Mean sub-token length (frequency-weighted) as a function of training hours for Latin-script languages (n=29). A positive correlation indicates that higher-resource languages tend to have slightly longer sub-tokens.