Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR
Yuxiang Mei, Dongxing Xu, Jiaen Liang, Yanhua Long
TL;DR
This work evaluates Speech-LLM approaches for multilingual conversational ASR by systematically comparing fine-tuning strategies for Whisper, exploring cross-attention based fusion of parallel Whisper and mHuBERT encoders, and testing two projector designs to connect to an LLM. It demonstrates that parameter-efficient fine-tuning of Whisper (e.g., LoRA) combined with a two-stage training regime yields strong in-domain performance, while more complex fusion mechanisms provide limited gains after joint LLM training. Direct comparisons with carefully tuned E2E Whisper baselines show that Speech-LLM approaches are competitive but still do not close the gap to fully fine-tuned E2E systems on the same data, underscoring ongoing design and optimization challenges. The findings offer practical guidance for Speech-LLM design, including when to favor LoRA, simple projectors, and gating-based fusion to balance performance and generalization, and they provide a publicly available codebase for reproducibility in multilingual ASR research.
Abstract
The INTERSPEECH 2025 Challenge on Multilingual Conversational Speech Language Models (MLC-SLM) promotes multilingual conversational ASR with large language models (LLMs). Our previous SHNU-mASR system adopted a competitive parallel-speech-encoder architecture that integrated Whisper and mHuBERT with an LLM. However, it faced two challenges: simple feature concatenation may not fully exploit complementary information, and the performance gap between LLM-based ASR and end-to-end(E2E) encoder-decoder ASR remained unexplored. In this work, we present an enhanced LLM-based ASR framework that combines fine-tuned Whisper and mHuBERT encoders with an LLM to enrich speech representations. We first evaluate E2E Whisper models with LoRA and full fine-tuning on the MLC-SLM ASR task, and then propose cross-attention-based fusion mechanisms for the parallel-speech-encoder. On the official evaluation set of the MLC-SLM Challenge, our system achieves a CER/WER of 10.69%, ranking on par with the top-ranked Track 1 systems, even though it uses only 1,500 hours of baseline training data compared with their large-scale training sets. Nonetheless, we find that our final LLM-based ASR still does not match the performance of a fine-tuned E2E Whisper model, providing valuable empirical guidance for future Speech-LLM design. Our code is publicly available at https://github.com/1535176727/MLC-SLM.