Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

TL;DR

The paper introduces Llama-SMoP, a sparse mixture of projectors module that scales multimodal LLM-based AVSR while keeping inference costs low. By employing MoE projectors with three routing configurations, particularly the Disjoint-Experts, Disjoint-Routers (DEDR) design, the approach improves ASR, VSR, and AVSR performance on LRS3, especially with smaller encoders and LLMs. Comprehensive ablations show that optimal performance arises with a modest number of experts and balanced routing, and that Llama-SMoP offers robust performance under noisy conditions and across model sizes. This work demonstrates a practical path to deploying efficient, high-performing AVSR systems in resource-constrained environments, with potential for integration alongside existing feature-fusion methods.

Abstract

Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy environments by integrating visual cues. While recent advances integrate Large Language Models (LLMs) into AVSR, their high computational cost hinders deployment in resource-constrained settings. To address this, we propose Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of Projectors (SMoP) module to scale model capacity without increasing inference costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors, Llama-SMoP enables the use of smaller LLMs while maintaining strong performance. We explore three SMoP configurations and show that Llama-SMoP DEDR (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation studies confirm its effectiveness in expert activation, scalability, and noise robustness.

Figures (5)

• Figure 1: Illustration of the overall framework of the proposed Llama-SMoP model, where audio and video tokens are embedded using a sparsely-gated mixture-of-experts scheme. and represent whether the module is trained or kept frozen.
• Figure 2: Detailed illustration of the three proposed SMoP configurations. (a) Joint-Experts, Joint-Router (JEJR) uses one multimodal router and one pool of expert for embedding audio-visual representations. (b) Disjoint-Experts, Disjoint-Routers (DEDR) uses modality-specific routers and experts for embedding modality-specific representations. (c) Joint-Experts, Disjoint-Routers (JEDR) uses modality-specific routers and one shared group of experts, so routers assign the Top-K experts considering the modality characteristics.
• Figure 3: (Left). ASR results for Llama-SMoP using different-size Whisper models with Llama 3.2-1B. (Middle). ASR results for Llama-SMoP using different-size Whisper models with Llama 3.2-3B. (Right). VSR results for Llama-SMoP with Llama 3.2-1B/3.2-3B.
• Figure 4: Ablation analysis on the number of expert projectors for the ASR and VSR tasks.
• Figure 5: Proportion of tokens assigned to each expert, either as first or second choice.