Speech Recognition Rescoring with Large Speech-Text Foundation Models

TL;DR

The paper tackles the challenge of improving ASR accuracy when transcribed data are scarce by introducing multi-modal Speech-Text Foundation Models for second-pass rescoring. It trains a decoder-only transformer on joint text and acoustic-unit tokens derived from a HuBERT encoder, enabling rescoring via $P_{LM}(Z)$ and $P_{MMLM}(z)$ with two sequence orders (speech-first and text-first). Discriminative fine-tuning using MWER is applied to optimize rescoring performance, with explicit objective $L_{mwer}(a, y^*) = \sum_{i=1}^N P(x_i|a) \epsilon(y_i, y^*)$ where $P(x_i|a) = \frac{e^{s_i}}{\sum_{j=1}^N e^{s_j}}$. Experiments on Librispeech, Tedlium, and out-of-domain datasets show up to 20% relative improvements over Whisper large and up to 15% over text-only LLMs, with notable cross-modal transfer advantages even when only one modality is present in testing.

Abstract

Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often limited by available transcribed speech data and benefit from a second pass rescoring using LLM. Recently multi-modal large language models, particularly speech and text foundational models have demonstrated strong spoken language understanding. Speech-Text foundational models leverage large amounts of unlabelled and labelled data both in speech and text modalities to model human language. In this work, we propose novel techniques to use multi-modal LLM for ASR rescoring. We also explore discriminative training to further improve the foundational model rescoring performance. We demonstrate cross-modal knowledge transfer in speech-text LLM can benefit rescoring. Our experiments demonstrate up-to 20% relative improvements over Whisper large ASR and up-to 15% relative improvements over text-only LLM.