Speech-Aware Long Context Pruning and Integration for Contextualized Automatic Speech Recognition

TL;DR

This work tackles the challenge of leveraging long-context information in contextualized ASR by introducing SAP2, a two-stage long-context pruning and integration framework that uses Speech-driven Attention-based Pooling to compress OCR-derived contextual keywords while preserving speech-relevant signals. By jointly training a context-pruning module and a SpeechLLM-based ASR system (SAP2-TPI), the approach achieves state-of-the-art WER on SlideSpeech ($7.71\%$ with five slides) and LibriSpeech ($1.12\%$), along with a $41.1\%$ relative reduction in biased keyword errors on SlideSpeech. The method demonstrates strong robustness to extended context lengths and scales effectively to long-form inputs, making it promising for domain-specific ASR tasks like conference presentations. Overall, SAP2 provides a practical, scalable mechanism to integrate long-context cues without sacrificing efficiency, enabling more accurate contextualized transcription in real-world settings.

Abstract

Automatic speech recognition (ASR) systems have achieved remarkable performance in common conditions but often struggle to leverage long-context information in contextualized scenarios that require domain-specific knowledge, such as conference presentations. This challenge arises primarily due to constrained model context windows and the sparsity of relevant information within extensive contextual noise. To solve this, we propose the SAP$^{2}$ method, a novel framework that dynamically prunes and integrates relevant contextual keywords in two stages. Specifically, each stage leverages our proposed Speech-Driven Attention-based Pooling mechanism, enabling efficient compression of context embeddings while preserving speech-salient information. Experimental results demonstrate state-of-the-art performance of SAP$^{2}$ on the SlideSpeech and LibriSpeech datasets, achieving word error rates (WER) of 7.71% and 1.12%, respectively. On SlideSpeech, our method notably reduces biased keyword error rates (B-WER) by 41.1% compared to non-contextual baselines. SAP$^{2}$ also exhibits robust scalability, consistently maintaining performance under extensive contextual input conditions on both datasets.

Figures (10)

• Figure 1: Comparisons of the previous SOTA MaLa-ASR, the baseline Qwen2-Audio-PC, and our SAP2-TPI on the SlideSpeech test set. For each sample, red texts indicate recognition errors in proper nouns, while green-highlighted texts showcase corrections made by SAP2-TPI. These two examples represent typical cases of proper noun recognition: the left demonstrates SAP2-TPI's accuracy in recognizing rare personal names, and the right highlights its capability in identifying medical terminology.
• Figure 2: The overall architecture of proposed SAP2-TPI framework: In stage one, long contextual keywords are pruned based on speech to reduce irrelevant information. Pruned contextual keywords are inputs of stage two, used for contextualized speech recognition. Speech-driven attention-based pooling is utilized in both stages to compress context embeddings.
• Figure 3: Distribution of tokenized keyword counts. (x-axis: number of tokens, linear scale; y-axis: instance count, log scale). Single-slide contexts exhibit mean/median token lengths of 64.57/53, while five-slide contexts reach 402.81/332.
• Figure 4: Analysis of the impact of pooling window sizes on WER. Experiments are conducted on L95 dataset (5 slides).
• Figure 5: WER across context lengths (1, 3, 5, 7, 9, 15, 25 slides) for SAP2-TPI and Qwen2-Audio-TPI, both trained on 5-slide segments (L95 dataset)