MiDashengLM: Efficient Audio Understanding with General Audio Captions

TL;DR

MiDashengLM addresses the need for open, general audio-language understanding by replacing ASR-centric pretraining with general audio captions via ACAVCaps and MECAT. The framework uses a Dasheng-based audio encoder aligned to text, followed by pretraining on public data and supervised fine-tuning with LoRA, enabling efficient, variable-length inputs and fast inference. The results show strong cross-domain performance: competitive to baselines on audio captioning, MECAT benchmarks, and paralinguistic tasks, with substantial inference speedups, while ASR performance lags behind closed models but remains robust across languages. The work emphasizes transparency and reproducibility through publicly available datasets and code.

Abstract

Current approaches for large audio language models (LALMs) often rely on closed data sources or proprietary models, limiting their generalization and accessibility. This paper introduces MiDashengLM, a novel open audio-language model designed for efficient and comprehensive audio understanding through the use of general audio captions using our novel ACAVCaps training dataset. MiDashengLM exclusively relies on publicly available pretraining and supervised fine-tuning (SFT) datasets, ensuring full transparency and reproducibility. At its core, MiDashengLM integrates Dasheng, an open-source audio encoder, specifically engineered to process diverse auditory information effectively. Unlike previous works primarily focused on Automatic Speech Recognition (ASR) based audio-text alignment, our strategy centers on general audio captions, fusing speech, sound and music information into one textual representation, enabling a holistic textual representation of complex audio scenes. Lastly, MiDashengLM provides an up to 4x speedup in terms of time-to-first-token (TTFT) and up to 20x higher throughput than comparable models. Checkpoints are available online at https://huggingface.co/mispeech/midashenglm-7b and https://github.com/xiaomi-research/dasheng-lm.

Figures (8)

• Figure 1: Training cross entropy loss (next token) curves between ASR and caption based pretraining. General captions utilize the ACAVCaps (\ref{['tab:env_data']}) dataset, while ASR uses ACAV100M-Speech (\ref{['tab:speech_data']}). ACAV100M-Speech contains up to 90 different languages, while captions are English only.
• Figure 2: Proposed MiDashengLM framework. For all three stages, training is done with standard next-token prediction loss. Stage 1 aligns the audio encoder with the text modality, after which the audio encoder is taken and initialized for Stage 2.
• Figure 3: Histrogram plot of training data sample lengths.
• Figure 4: Our proposed data curation pipeline. We filter ACAV100M with an automatic pipeline, that predicts transcripts, sound events, sound quality and other meta information. A reasoning-LLM is then used to generate a caption from the provided meta information. The resulting curated dataset is then split into a training set (ACAVCaps) and a novel evaluation set Multi-Expert Chain for Audio Tasks (MECAT).
• Figure 5: Subtasks of the proposed MECAT-QA testset.