SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation

TL;DR

The paper addresses the challenge of reproducible, fair ASR evaluation by introducing the SpeechColab Leaderboard, an open-source platform with a Dataset Zoo, Model Zoo, and an end-to-end Evaluation Pipeline. It analyzes scoring subtleties and demonstrates how pipeline choices can dramatically affect benchmark results, prompting the introduction of the Modified Token Error Rate ($mTER$), a symmetric and normalized alternative to $TER$ built on Levenshtein distance and inspired by Kolmogorov complexity notions like Normalized Information Distance. Empirically, $mTER$ proves robust and backward-compatible with $TER$ across large-scale benchmarks, while providing well-behaved metric properties essential for cross-study comparisons. The platform and metric together offer a practical path toward fair comparisons of open-source and industrial ASR systems and invite broader community participation and dataset/model expansion.

Abstract

In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes. These include nuances related to capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of the transition towards an End-to-End future. (iii) We propose a practical modification to the conventional Token-Error-Rate (TER) evaluation metric, with inspirations from Kolmogorov complexity and Normalized Information Distance (NID). This adaptation, called modified-TER (mTER), achieves proper normalization and symmetrical treatment of reference and hypothesis. By leveraging this platform as a large-scale testing ground, this study demonstrates the robustness and backward compatibility of mTER when compared to TER. The SpeechColab Leaderboard is accessible at https://github.com/SpeechColab/Leaderboard

Figures (5)

• Figure 1: SpeechColab Leaderboard
• Figure 2: Levenshtein distance with Dynamic Alternative Expansion (DAE)
• Figure 3: Results of ablation experiments on GigaSpeech.test dataset
• Figure 4: An example showing the difference in values between mTER and TER.
• Figure 5: TER vs mTER scatter plots