Multi-stage Large Language Model Correction for Speech Recognition

TL;DR

Addresses ASR error correction with LLMs by gating corrections behind an uncertainty check and applying a rule-based, multi-step LLM prompting strategy in zero-shot mode. The method computes a combined score for N-best hypotheses, Score($y_i$) = log P_ASR($y_i$|$x$) + α log P_LM($y_i$), then uses Softmax and a threshold to decide Stage2. Stage2 uses a single, rule-driven prompt that constrains outputs to words from the N-best list, limits creativity, standardizes punctuation, and omits explanations. Across LibriSpeech, Common Voice, and MLS, the approach yields 10-20% relative WER improvements and 1.3% WER on LibriSpeech test-clean, approaching state-of-the-art in zero-shot settings and offering practical gains for diverse ASR deployments.

Abstract

In this paper, we investigate the usage of large language models (LLMs) to improve the performance of competitive speech recognition systems. Different from previous LLM-based ASR error correction methods, we propose a novel multi-stage approach that utilizes uncertainty estimation of ASR outputs and reasoning capability of LLMs. Specifically, the proposed approach has two stages: the first stage is about ASR uncertainty estimation and exploits N-best list hypotheses to identify less reliable transcriptions; The second stage works on these identified transcriptions and performs LLM-based corrections. This correction task is formulated as a multi-step rule-based LLM reasoning process, which uses explicitly written rules in prompts to decompose the task into concrete reasoning steps. Our experimental results demonstrate the effectiveness of the proposed method by showing 10% ~ 20% relative improvement in WER over competitive ASR systems -- across multiple test domains and in zero-shot settings.

Figures (1)

• Figure 1: Overview of the multi-stage correction pipeline. The first stage is to detect less reliable transcriptions while the second stage performs a LLM-based correction. A confidence threshold will decide if transcriptions need to be sent to the second stage.