Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

TL;DR

The paper tackles robust speech recognition by enabling large language models to denoise ASR outputs in a noise-aware generative error correction framework. It introduces RobustHyPoradise, a large dataset of noisy ASR hypotheses and ground-truth transcriptions, and proposes language-space noise embeddings derived from N-best hypothesis diversity, paired with mutual-information-based audio noise distillation. The method integrates these embeddings into an LLM adapter (RobustGER) to map $N$-best hypotheses to clean transcriptions, achieving up to $53.9\%$ relative WER improvement with limited training data and across various noise conditions. This approach bypasses cross-modality alignment issues by operating in the language space, offering data-efficient, scalable improvements for real-world noisy ASR with several state-of-the-art LLM backbones. The work also provides extensive ablations and analyses, including embedding visualizations and data-efficiency studies, highlighting the practical impact of language-space denoising for GER in noisy environments.

Abstract

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recognition results. The latest work proposes a GER benchmark with HyPoradise dataset to learn the mapping from ASR N-best hypotheses to ground-truth transcription by efficient LLM finetuning, which shows great effectiveness but lacks specificity on noise-robust ASR. In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER just like what robust ASR do}, where one solution is introducing noise information as a conditioner into LLM. However, directly incorporating noise embeddings from audio encoder could harm the LLM tuning due to cross-modality gap. To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER. Furthermore, in order to enhance its representation ability of audio noise, we design a knowledge distillation (KD) approach via mutual information estimation to distill the real noise information in audio embeddings to our language embedding. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate while with limited training data. Analysis shows that our language-space noise embedding can well represent the noise conditions of source speech, under which off-the-shelf LLMs show strong ability of language-space denoising.

Figures (6)

• Figure 1: Overview of (a) GER chen2023hpyang2023generative, (b) GER with audio-space denoising zhang2023llama (see details in §\ref{['assec:denoise_llm_tuning']}), (c) GER with language-space denoising.
• Figure 2: Left: The RobustGER framework that leverages efficient LLM finetuning to learn mapping from ASR N-best hypotheses to ground-truth transcription, where we propose a language-space noise embedding with audio distillation to denoise GER process. Right: The extraction of language-space noise embedding from N-best hypotheses by measuring its diversity, where we calculate the utterance- and token-level embedding differences between each pair of hypotheses in the N-best list. The details of embedding extraction are illustrated in §\ref{['ssec:noise_emb']} and Eq. (\ref{['eq4']})-(\ref{['eq6']}).
• Figure 3: Audio noise distillation by mutual information neural estimation (MINE). The trainable tuner is designed to maximize the MI between our extracted noise embedding and the noisy speech.
• Figure 4: t-SNE visualizations of (a) language-space noise embedding, (b) language embedding with audio distillation, (c) audio noise embeddings. Cluster distances are in Table \ref{['table:tsne_distance']}. Details are in §\ref{['asssec:details_tsne']}.
• Figure 5: LLaMA-Adapter tuning zhang2023llama with language-space denoising (ours).