Fast Word Error Rate Estimation Using Self-Supervised Representations for Speech and Text
Chanho Park, Chengsong Lu, Mingjie Chen, Thomas Hain
TL;DR
Fe-WER addresses the need for fast WER estimation without ground-truth transcripts by a two-tower architecture that encodes speech and text with self-supervised representations, applies average pooling, and predicts $\widehat{\text{WER}}$ via an MLP. The model optimizes $\text{MSE} = \frac{1}{N} \sum_i (\text{WER}_i - \widehat{\text{WER}}_i)^2$ and supports a duration-weighted variant for unlabeled data. It substantially outperforms a BiLSTM baseline (14.10% RMSE reduction and 1.22% PCC gain on TL3) while delivering a 3.4x faster inference speed, with HuBERT and XLM-R identified as the strongest SSLR pairing. The approach enables scalable WER assessment for large-scale ASR systems, informing data curation and self-training pipelines without ground-truth labels.
Abstract
Word error rate (WER) estimation aims to evaluate the quality of an automatic speech recognition (ASR) system's output without requiring ground-truth labels. This task has gained increasing attention as advanced ASR systems are trained on large amounts of data. In this context, the computational efficiency of a WER estimator becomes essential in practice. However, previous works have not prioritised this aspect. In this paper, a Fast estimator for WER (Fe-WER) is introduced, utilizing average pooling over self-supervised learning representations for speech and text. Our results demonstrate that Fe-WER outperformed a baseline relatively by 14.10% in root mean square error and 1.22% in Pearson correlation coefficient on Ted-Lium3. Moreover, a comparative analysis of the distributions of target WER and WER estimates was conducted, including an examination of the average values per speaker. Lastly, the inference speed was approximately 3.4 times faster in the real-time factor.