CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR
Muhammad Shakeel, Yosuke Fukumoto, Chikara Maeda, Chyi-Jiunn Lin, Shinji Watanabe
TL;DR
The paper tackles the challenge of personalized, multi-speaker ASR under overlapping speech by fusing target-speaker conditioning with contextual biasing in a unified end-to-end framework. CALM combines target-speaker embeddings (via a speaker encoder and FiLM modulation) with a BiasEnc module that converts biasing phrases into a dynamic vocabulary, jointly optimized with CTC, interCTC, and attention losses plus a VAD auxiliary loss. Across LibriSpeechMix, CSJMix, and AMI, CALM achieves substantial reductions in biased-word error rates (B-WER/B-CER) and improvements in overall WER/CER, demonstrating cross-lingual generalization and robustness to varying bias list sizes. The approach significantly advances personalized multi-speaker ASR by leveraging the inter-dependencies between acoustic and linguistic contexts, enabling scalable, speaker-aware transcription in realistic conversational settings.
Abstract
We present CALM, a joint Contextual Acoustic-Linguistic Modeling framework for multi-speaker automatic speech recognition (ASR). In personalized AI scenarios, the joint availability of acoustic and linguistic cues naturally motivates the integration of target-speaker conditioning with contextual biasing in overlapping conversations. CALM implements this integration in an end-to-end framework through speaker embedding-driven target-speaker extraction and dynamic vocabulary-based contextual biasing. We evaluate CALM on simulated English (LibriSpeechMix) and Japanese (Corpus of Spontaneous Japanese mixtures, CSJMix). On two-speaker mixtures, CALM reduces biased word error rate (B-WER) from 12.7 to 4.7 on LibriSpeech2Mix and biased character error rate (B-CER) from 16.6 to 8.4 on CSJMix2 (eval3), demonstrating the effectiveness of joint acoustic-linguistic modeling across languages. We additionally report results on the AMI corpus (IHM-mix condition) to validate performance on standardized speech mixtures.