U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding

TL;DR

U-SAM tackles the fragmentation of audio understanding across speech, audio events, and music by unifying three domain-specialized encoders with a pre-trained LLM. It introduces a Task-Aware Projection Module (TAPM) built on a Mixture of Experts to adaptively fuse features according to the task, and a Semantic-Aware Contrastive Loss (SACLM) to refine cross-modal alignment by highlighting semantically relevant frames. The approach uses LoRA to efficiently fine-tune the LLM and a Window-level Q-Former to handle variable-length inputs, achieving strong results across ASR, speech translation, audio captioning, and music captioning, with emergent capabilities on unseen tasks. The combination yields superior cross-domain performance and practical potential for generalized audio-language understanding, with code available at the provided repository.

Abstract

The text generation paradigm for audio tasks has opened new possibilities for unified audio understanding. However, existing models face significant challenges in achieving a comprehensive understanding across diverse audio types, such as speech, general audio events, and music. Furthermore, their exclusive reliance on cross-entropy loss for alignment often falls short, as it treats all tokens equally and fails to account for redundant audio features, leading to weaker cross-modal alignment. To deal with the above challenges, this paper introduces U-SAM, an advanced audio language model that integrates specialized encoders for speech, audio, and music with a pre-trained large language model (LLM). U-SAM employs a Mixture of Experts (MoE) projector for task-aware feature fusion, dynamically routing and integrating the domain-specific encoder outputs. Additionally, U-SAM incorporates a Semantic-Aware Contrastive Loss Module, which explicitly identifies redundant audio features under language supervision and rectifies their semantic and spectral representations to enhance cross-modal alignment. Extensive experiments demonstrate that U-SAM consistently outperforms both specialized models and existing audio language models across multiple benchmarks. Moreover, it exhibits emergent capabilities on unseen tasks, showcasing its generalization potential. Code is available (https://github.com/Honee-W/U-SAM/).

Figures (2)

• Figure 1: (a) U-SAM architecture: Integrates three encoders (Whisper, CED, and MERT) to extract multi-scale audio features, refined by a Q-Former and Task-Aware Projection Module (TAPM) for task-specific adaptation. The adapted features, combined with text prompts, are processed by a large language model (LLM) for semantic understanding and response generation. (b) Semantic-Aware Contrastive Loss Module (SACLM): Predicts significance scores for audio embeddings via the Score-Aware Prediction Network, generates a decision matrix for frame selection, refines frames with an Aggregation Network, and aligns audio embeddings with text using a Triplet Loss.
• Figure 2: Task-Aware Projection Module (TAPM). TAPM uses a Mixture of Experts to dynamically project audio embeddings based on routing weights derived from text prompt embeddings. The weighted outputs of $n$ experts are combined to generate task-adaptive features for the downstream LLM.