Audio-Language Models for Audio-Centric Tasks: A survey

TL;DR

Audio-Language Models (ALMs) tackle audio-centric tasks by leveraging natural language supervision, enabling zero-shot generalization and flexible downstream transfer. The survey organizes ALMs around architectures (Two Towers, Two Heads, One Head, Cooperated Systems), training objectives (contrastive, generative, discriminative), and evaluation regimes (zero-shot, linear probe, fine-tuning, instruction-following), and then connects these to representation pre-training (CNN/Transformer/codec), language pre-training, and audio–language pre-training approaches. It then synthesizes downstream transfer strategies, including task-specific fine-tuning, multi-task tuning, and agent-based systems, and catalogs datasets and benchmarks spanning audio captions, transcription, QA, and cross-task evaluation. The paper also discusses key challenges—data diversity, unified encoders, continual and data-efficient learning, and evaluation standardization—and offers a roadmap for future ALM development with practical implications for real-world audio understanding and generation tasks.

Abstract

Audio-Language Models (ALMs), which are trained on audio-text data, focus on the processing, understanding, and reasoning of sounds. Unlike traditional supervised learning approaches learning from predefined labels, ALMs utilize natural language as a supervision signal, which is more suitable for describing complex real-world audio recordings. ALMs demonstrate strong zero-shot capabilities and can be flexibly adapted to diverse downstream tasks. These strengths not only enhance the accuracy and generalization of audio processing tasks but also promote the development of models that more closely resemble human auditory perception and comprehension. Recent advances in ALMs have positioned them at the forefront of computer audition research, inspiring a surge of efforts to advance ALM technologies. Despite rapid progress in the field of ALMs, there is still a notable lack of systematic surveys that comprehensively organize and analyze developments. In this paper, we present a comprehensive review of ALMs with a focus on general audio tasks, aiming to fill this gap by providing a structured and holistic overview of ALMs. Specifically, we cover: (1) the background of computer audition and audio-language models; (2) the foundational aspects of ALMs, including prevalent network architectures, training objectives, and evaluation methods; (3) foundational pre-training and audio-language pre-training approaches; (4) task-specific fine-tuning, multi-task tuning and agent systems for downstream applications; (5) datasets and benchmarks; and (6) current challenges and future directions. Our review provides a clear technical roadmap for researchers to understand the development and future trends of existing technologies, offering valuable references for implementation in real-world scenarios.

Figures (11)

• Figure 1: A timeline of recent advances in audio-language models. Is is established mainly according to the release date (e.g., the submission date to arXiv) and some still working in progress. It highlights that datasets serve as the foundation for inspiring research in pre-training and downstream models. With the advancement of model research, recent studies have developed several benchmarks to promote comprehensive development in the field.
• Figure 2: Research landscape for audio-language models. From the perspective of model training, (a) audio-language representation requires pre-training (Sec. \ref{['section: Pre-training']}), (b) transfer to downstream application through task-specific fine-tuning or instruction tuning (Sec. \ref{['section: transfer']}), (c) data is the foundation for model training, and they can be divided into labeled audio datasets, audio-text paired datasets, and audio question answering datasets (Sec. \ref{['section: Data']}).
• Figure 3: Research outline on audio-language models for audio-centric tasks.
• Figure 4: Typical architectures of audio-language models. (a) Two Towers, with one encoder and a projector for each modality, embeddings will be aligned in a joint space. (b) Two Heads, adds language model on top. (c) One Head, with one unified encoder and a language model. (d) Cooperated Systems, utilize LLMs as agents to cooperate several models.
• Figure 5: Illustration of audio-language models training objectives. (a) Pre-training objectives include contrastive, generative, and discriminative objectives, which may be conducted on audio-text or single-modal data. The transfer objectives can be (b) task-specific fine-tuning objectives or (c) generative language modeling objective.