ALARM: Audio-Language Alignment for Reasoning Models

TL;DR

This work proposes self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment and achieving the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.

Abstract

Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.

Figures (3)

• Figure 1: Overview of the corpora collection pipeline.
• Figure 2: Overview of ALARM training pipeline. A pre-trained RLM first generates an initial response $R_0$ from textual metadata, then rephrases it into $R_{\text{text}}$. Finally, the RLM is equipped with the audio-fusion module and trained to generate $R_{\text{text}}$ from the corresponding audio inputs.
• Figure 3: Comparison of our models on AIR-Bench benchmark across different tasks.