asr_eval: Algorithms and tools for multi-reference and streaming speech recognition evaluation

TL;DR

asr_eval tackles the challenge of evaluating speech recognition under multi-reference annotations and streaming conditions. It introduces MWER, a multi-reference aware alignment algorithm, and a Python library that provides dashboards, model wrappers, and streaming evaluation tools. The authors also release DiverseSpeech-Ru and perform multi-reference relabeling of Sova-RuDevices to study fine-tuning dynamics, demonstrating that dataset labeling style can create metric artefacts if normalization is used alone. The work provides practical tooling and datasets to enable robust benchmarking, particularly for long-form and non-Latin languages, and to mitigate dataset-specific labeling bias.

Abstract

We propose several improvements to the speech recognition evaluation. First, we propose a string alignment algorithm that supports both multi-reference labeling, arbitrary-length insertions and better word alignment. This is especially useful for non-Latin languages, those with rich word formation, to label cluttered or longform speech. Secondly, we collect a novel test set DiverseSpeech-Ru of longform in-the-wild Russian speech with careful multi-reference labeling. We also perform multi-reference relabeling of popular Russian tests set and study fine-tuning dynamics on its corresponding train set. We demonstrate that the model often adopts to dataset-specific labeling, causing an illusion of metric improvement. Based on the improved word alignment, we develop tools to evaluate streaming speech recognition and to align multiple transcriptions to compare them visually. Additionally, we provide uniform wrappers for many offline and streaming speech recognition models. Our code will be made publicly available.

Figures (7)

• Figure 1: Multiple transcription alignment.
• Figure 2: Time remapping in streaming evaluation. X axis is a real time, and Y axis is a chunk number (the audio consists of 5 chunks). For each chunk, we mark send time (black dot) and time span when the system was busy processing this chunk (red). Left: all chunks were send at once. Right: all chunks were sent were sent evenly in real time. Blue denote idle time we want to eliminate. Time remapping takes chunk history from the process on the left and alters the recorded timestamps to mimic process on the right. Thus we get the real time simulations faster than real time.
• Figure 3: A streaming evaluation diagram. Each row is a partial alignment, where correctly transcribed words are show in green, wrongly transcribed words are shown in red, insertions (extra words) are shown as red dots (their timestamps are fictitious), deletions (missed words) and words not yet transcribed are shown in gray.
• Figure 4: A streaming quality historgram. See Section \ref{['streaming-evaluation']} for description.
• Figure 5: Upper, left. Does multivariant re-labeling of the test set alter the visible fine-tuning dynamics? We avaluate the same fine-tuning checkpoint series 4 times on the same test set, but with different labeling, and observe different dynamics. If we stop after step 800, for whisper-medium we get around 3% WER improvement for the original sova-rudevices test part with any normalizer, but 0% WER improvement for the multivariant one.