SeMaScore : a new evaluation metric for automatic speech recognition tasks

TL;DR

Overall, it is demonstrated that SeMaScore serves as a more dependable evaluation metric, particularly in real-world situations involving atypical speech patterns, particularly in real-world situations involving atypical speech patterns.

Abstract

In this study, we present SeMaScore, generated using a segment-wise mapping and scoring algorithm that serves as an evaluation metric for automatic speech recognition tasks. SeMaScore leverages both the error rate and a more robust similarity score. We show that our algorithm's score generation improves upon the state-of-the-art BERTScore. Our experimental results show that SeMaScore corresponds well with expert human assessments, signal-to-noise ratio levels, and other natural language metrics. We outperform BERTScore by 41x in metric computation speed. Overall, we demonstrate that SeMaScore serves as a more dependable evaluation metric, particularly in real-world situations involving atypical speech patterns.

Figures (6)

• Figure 1: Distribution of various metrics for disordered speech
• Figure 2: Correlation between SeMaScore and WER for disordered speech
• Figure 3: Correlation between SeMaScore and BERTScore for disordered speech
• Figure 4: Distribution of various metrics for noisy speech
• Figure 5: Correlation between SeMaScore and WER for noisy speech