MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free

TL;DR

The paper tackles gradient conflicts when extending LLMs to audio by introducing a sparse Mixture-of-Experts (MoE) Adapter that decouples heterogeneous acoustic attributes. By routing audio tokens to specialized experts while retaining shared components, the MoE-Adapter achieves better knowledge reasoning and paralinguistic understanding without increasing computational cost compared to dense baselines. Empirical results on a 1.7B backbone show consistent gains across MMSU, OBQA, and MMAU, and analyses reveal improved gradient alignment and constructive transfer between modalities. This approach offers a scalable pathway to more capable multimodal foundation models with efficient inference and interpretable expert specializations.

Abstract

Extending the input modality of Large Language Models~(LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acoustic information is intrinsically \textit{heterogeneous}, entangling attributes such as speech, music, and environmental context. Existing research is limited to a dense, parameter-shared adapter to model these diverse patterns, which induces \textit{gradient conflict} during optimization, as parameter updates required for distinct attributes contradict each other. To address this limitation, we introduce the \textit{\textbf{MoE-Adapter}}, a sparse Mixture-of-Experts~(MoE) architecture designed to decouple acoustic information. Specifically, it employs a dynamic gating mechanism that routes audio tokens to specialized experts capturing complementary feature subspaces while retaining shared experts for global context, thereby mitigating gradient conflicts and enabling fine-grained feature learning. Comprehensive experiments show that the MoE-Adapter achieves superior performance on both audio semantic and paralinguistic tasks, consistently outperforming dense linear baselines with comparable computational costs. Furthermore, we will release the related code and models to facilitate future research.

Figures (5)

• Figure 1: Visualization of Gradient Conflict. The naturally disjoint manifolds of Speech, Music, and Sound induce divergent gradient update directions ($\nabla \mathcal{L}$), creating an optimization conflict that pulls the shared adapter in contradictory ways.
• Figure 2: The overall architecture of the MoE Adapter compared with the Dense Adapter baseline. Unlike the monolithic, parameter-shared design in (a), the MoE-Adapter in (b) employs a sparse gating mechanism to dynamically route audio tokens to specialized experts. This architecture facilitates fine-grained feature learning and effectively mitigates gradient conflicts by disentangling heterogeneous acoustic attributes during the projection phase.
• Figure 3: Expert Activation Heatmaps. The expert balance loss prevents collapse, fostering balanced utilization while maintaining modality-aware specialization. Notably, Sound serves as an "Acoustic Bridge" sharing experts with both Speech and Music.
• Figure 4: Gradient Cosine Similarity. The dense FFN adapter (left) exhibits negative correlations between heterogeneous audio tasks, whereas the MoE adapter (right) demonstrates improved gradient alignment, indicating reduced optimization conflicts.
• Figure 5: Gradient Influence Analysis. Values represent the impact of updating task $j$ (x-axis) on task $i$ (y-axis). Unlike the FFN baseline (left) which suffers from harmful interference (negative values), the MoE adapter (right) promotes constructive transfer (positive values), effectively mitigating cross-task conflicts.