Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MATS: An Audio Language Model under Text-only Supervision

Wen Wang, Ruibing Hou, Hong Chang, Shiguang Shan, Xilin Chen

TL;DR

MATS tackles the high data and compute costs of large audio-language models by training a multimodal LLM with text-only supervision. It leverages CLAP’s shared audio-language embedding space, learning a mapper to project language descriptions into the LLM latent space, while using Gaussian noise during training and the Santa modality-transfer mechanism at inference to bridge audio-language gaps. A theoretical generalization bound ties model performance to empirical risk, modality discrepancy, and hypothesis-space complexity, motivating the Santa design. Empirically, MATS achieves competitive zero-shot and open-ended performance across diverse audio tasks, using AudioTIA-5M to enable broad task coverage, and demonstrates substantial efficiency compared to audio-supervised baselines. The work provides code and a scalable pathway toward cost-effective, versatile audio-language understanding in LLMs.

Abstract

Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. However, the training of LALMs demands a large corpus of audio-language pairs, which requires substantial costs in both data collection and training resources. In this paper, we propose \textbf{MATS}, an audio-language multimodal LLM designed to handle \textbf{M}ultiple \textbf{A}udio task using solely \textbf{T}ext-only \textbf{S}upervision. By leveraging pre-trained audio-language alignment models such as CLAP, we develop a text-only training strategy that projects the shared audio-language latent space into LLM latent space, endowing the LLM with audio comprehension capabilities without relying on audio data during training. To further bridge the modality gap between audio and language embeddings within CLAP, we propose the \textbf{S}trongly-rel\textbf{a}ted \textbf{n}oisy \textbf{t}ext with \textbf{a}udio (\textbf{Santa}) mechanism. Santa maps audio embeddings into CLAP language embedding space while preserving essential information from the audio input. Extensive experiments demonstrate that MATS, despite being trained exclusively on text data, achieves competitive performance compared to recent LALMs trained on large-scale audio-language pairs. The code is publicly available in \href{https://github.com/wangwen-banban/MATS}{https://github.com/wangwen-banban/MATS}.

MATS: An Audio Language Model under Text-only Supervision

TL;DR

MATS tackles the high data and compute costs of large audio-language models by training a multimodal LLM with text-only supervision. It leverages CLAP’s shared audio-language embedding space, learning a mapper to project language descriptions into the LLM latent space, while using Gaussian noise during training and the Santa modality-transfer mechanism at inference to bridge audio-language gaps. A theoretical generalization bound ties model performance to empirical risk, modality discrepancy, and hypothesis-space complexity, motivating the Santa design. Empirically, MATS achieves competitive zero-shot and open-ended performance across diverse audio tasks, using AudioTIA-5M to enable broad task coverage, and demonstrates substantial efficiency compared to audio-supervised baselines. The work provides code and a scalable pathway toward cost-effective, versatile audio-language understanding in LLMs.

Abstract

Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. However, the training of LALMs demands a large corpus of audio-language pairs, which requires substantial costs in both data collection and training resources. In this paper, we propose \textbf{MATS}, an audio-language multimodal LLM designed to handle \textbf{M}ultiple \textbf{A}udio task using solely \textbf{T}ext-only \textbf{S}upervision. By leveraging pre-trained audio-language alignment models such as CLAP, we develop a text-only training strategy that projects the shared audio-language latent space into LLM latent space, endowing the LLM with audio comprehension capabilities without relying on audio data during training. To further bridge the modality gap between audio and language embeddings within CLAP, we propose the \textbf{S}trongly-rel\textbf{a}ted \textbf{n}oisy \textbf{t}ext with \textbf{a}udio (\textbf{Santa}) mechanism. Santa maps audio embeddings into CLAP language embedding space while preserving essential information from the audio input. Extensive experiments demonstrate that MATS, despite being trained exclusively on text data, achieves competitive performance compared to recent LALMs trained on large-scale audio-language pairs. The code is publicly available in \href{https://github.com/wangwen-banban/MATS}{https://github.com/wangwen-banban/MATS}.

Paper Structure

This paper contains 26 sections, 1 theorem, 12 equations, 10 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Formulations
  5. Model Architecture
  6. Theoretical Analysis on Generalization
  7. Santa Mechanism
  8. Training Pipeline
  9. Experiment
  10. Experimental Setup
  11. Main Results
  12. Close-ended Results
  13. Open-ended Results
  14. Ablation
  15. Conclusion
  16. ...and 11 more sections

Key Result

Theorem 3.1

Let $\mathcal{H}$ be a hypothesis space of Natarajan-dimension $d$. For classification with $V$ classes, let $\mathcal{D}_{tr}$ be text-only training set drawn from distribution $p_{\mathcal{T}}\left(z^t, y\right)$. As for test set with audio files draw from distribution $p_{\mathcal{A}}\left(z^a, y Here, $\mathrm{disc}_{L_1}$ denotes the Discrepancy Distance (Definition 4 in mansour2023domainadap

Figures (10)

  • Figure 1: Performance of MATS compared to the previous SOTA chu2023qwenaudioadvancinguniversalaudiokong2024audioflamingonovelaudiodeshmukh2024pengiaudiolanguagemodel under zero-shot setting for close-ended audio tasks.
  • Figure 2: The architecture of proposed MATS.
  • Figure 3: t-SNE visualizations for various methods on 350 randomly selected language embeddings (blue) and paired audio embeddings (red) from training dataset of MATS-Audio in \ref{['tab:train dataset audio']}. a) CLAP b) CLAP with Gaussian Noise deshmukh2024training c) CLAP with memory bank li2024drcapdecodingclaplatents d) CLAP with Santa.
  • Figure S4: Pipeline And Prompt of generating audio descriptions.
  • Figure S5: The effect of different $\lambda$ on AudioCaps.
  • ...and 5 more figures

Theorems & Definitions (2)

  • Theorem 3.1
  • proof