Language Family Matters: Evaluating LLM-Based ASR Across Linguistic Boundaries

TL;DR

The paper tackles multilingual ASR with SpeechLLMs by bridging frozen speech encoders to LLMs via lightweight connectors. It introduces a family-based connector strategy, training one adapter per language family rather than per language, and evaluates across two LLM backbones, two corpora, and ten families. The results show family connectors generally reduce parameter counts and improve generalization, especially under cross-domain transfer, though some highly diverse families face mixed outcomes. The findings advocate a scalable, linguistically informed approach to deploying multilingual SpeechLLMs with robust cross-domain performance.

Abstract

Large Language Model (LLM)-powered Automatic Speech Recognition (ASR) systems achieve strong performance with limited resources by linking a frozen speech encoder to a pretrained LLM via a lightweight connector. Prior work trains a separate connector per language, overlooking linguistic relatedness. We propose an efficient and novel connector-sharing strategy based on linguistic family membership, enabling one connector per family, and empirically validate its effectiveness across two multilingual LLMs and two real-world corpora spanning curated and crowd-sourced speech. Our results show that family-based connectors reduce parameter count while improving generalization across domains, offering a practical and scalable strategy for multilingual ASR deployment.

Figures (1)

• Figure 1: Overall framework for multilingual ASR.