Pisets: A Robust Speech Recognition System for Lectures and Interviews

TL;DR

Pisets addresses robust offline transcription of lectures and interviews by reducing hallucinations in autoregressive ASR. The approach stacks Wav2Vec2 for speech segment detection, AST for false positive filtering, and Whisper for final transcription, with curriculum learning on Russian corpora and uncertainty modeling to boost domain adaptability. The paper reports improved lexical and semantic quality on long Russian audio with noisy conditions, and proposes uncertainty metrics such as token scores and model disagreement to guide human in the loop. The work demonstrates practical benefits for researchers and journalists and provides open source implementations under Apache 2.0.

Abstract

This work presents a speech-to-text system "Pisets" for scientists and journalists which is based on a three-component architecture aimed at improving speech recognition accuracy while minimizing errors and hallucinations associated with the Whisper model. The architecture comprises primary recognition using Wav2Vec2, false positive filtering via the Audio Spectrogram Transformer (AST), and final speech recognition through Whisper. The implementation of curriculum learning methods and the utilization of diverse Russian-language speech corpora significantly enhanced the system's effectiveness. Additionally, advanced uncertainty modeling techniques were introduced, contributing to further improvements in transcription quality. The proposed approaches ensure robust transcribing of long audio data across various acoustic conditions compared to WhisperX and the usual Whisper model. The source code of "Pisets" system is publicly available at GitHub: https://github.com/bond005/pisets.

Figures (3)

• Figure 1: Proposed three-component speech recognition architecture
• Figure 2: The error detection recall and uncertainty ratio of different uncertainty estimation methods. The results are averaged across 7 long Russian audios, and the results for individual audios are shown in semi-transparent. Whisper scores method is show as a line for different score threshold. All model disagreement and ensembling methods cannot reliably outperform Whisper scores as a source of uncertainty. It can be seen that if we mark only around 5% words as uncertain, we can accumulate in them 35% of all errors (excluding errors caused by missed words in transcription).
• Figure 3: The example of highlighting dubious places in the transcription, based on uncertainty estimation with model disagreement.