Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding

Jiaming Zhou, Xuxin Cheng, Shiwan Zhao, Yuhang Jia, Cao Liu, Ke Zeng, Xunliang Cai, Yong Qin

TL;DR

DIFFA-2 presents a strengthened diffusion-based large audio language model designed for general audio understanding. It uses a frozen speech encoder plus dual semantic and acoustic adapters, integrated into a diffusion LLM backbone, and trained via a four-stage curriculum that includes semantic-acoustic alignment, supervised fine-tuning, and variance-reduced preference optimization. With open-source data and a lightweight training footprint (approximately 1.1% of parameters trainable), DIFFA-2 achieves competitive results on MMSU, MMAU, and MMAR benchmarks, often surpassing first-generation diffusion baselines and approaching autoregressive LALMs under practical budgets. The work demonstrates that diffusion backbones are a viable path for scalable, structured audio understanding, while highlighting opportunities to further close gaps in dialogue-oriented tasks through targeted alignment data and tuning.

Abstract

Autoregressive (AR) large audio language models (LALMs) such as Qwen-2.5-Omni have achieved strong performance on audio understanding and interaction, but scaling them remains costly in data and computation, and strictly sequential decoding limits inference efficiency. Diffusion large language models (dLLMs) have recently been shown to make effective use of limited training data, and prior work on DIFFA indicates that replacing an AR backbone with a diffusion counterpart can substantially improve audio understanding under matched settings, albeit at a proof-of-concept scale without large-scale instruction tuning, preference alignment, or practical decoding schemes. We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding. DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum that combines semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization, using only fully open-source corpora. Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA and is competitive to strong AR LALMs under practical training budgets, supporting diffusion-based modeling is a viable backbone for large-scale audio understanding. Our code is available at https://github.com/NKU-HLT/DIFFA.git.

DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding

TL;DR

DIFFA-2 presents a strengthened diffusion-based large audio language model designed for general audio understanding. It uses a frozen speech encoder plus dual semantic and acoustic adapters, integrated into a diffusion LLM backbone, and trained via a four-stage curriculum that includes semantic-acoustic alignment, supervised fine-tuning, and variance-reduced preference optimization. With open-source data and a lightweight training footprint (approximately 1.1% of parameters trainable), DIFFA-2 achieves competitive results on MMSU, MMAU, and MMAR benchmarks, often surpassing first-generation diffusion baselines and approaching autoregressive LALMs under practical budgets. The work demonstrates that diffusion backbones are a viable path for scalable, structured audio understanding, while highlighting opportunities to further close gaps in dialogue-oriented tasks through targeted alignment data and tuning.

Abstract

Autoregressive (AR) large audio language models (LALMs) such as Qwen-2.5-Omni have achieved strong performance on audio understanding and interaction, but scaling them remains costly in data and computation, and strictly sequential decoding limits inference efficiency. Diffusion large language models (dLLMs) have recently been shown to make effective use of limited training data, and prior work on DIFFA indicates that replacing an AR backbone with a diffusion counterpart can substantially improve audio understanding under matched settings, albeit at a proof-of-concept scale without large-scale instruction tuning, preference alignment, or practical decoding schemes. We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding. DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum that combines semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization, using only fully open-source corpora. Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA and is competitive to strong AR LALMs under practical training budgets, supporting diffusion-based modeling is a viable backbone for large-scale audio understanding. Our code is available at https://github.com/NKU-HLT/DIFFA.git.
Paper Structure (47 sections, 8 equations, 6 figures, 11 tables)

This paper contains 47 sections, 8 equations, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methods
  4. Model Architecture
  5. Training Data Overview
  6. Transcription Data for Semantic Alignment.
  7. SFT Data.
  8. Preference Alignment Data.
  9. Progressive Four-Stage Training
  10. Stage 1: Semantic alignment on ASR.
  11. Stage 2: Joint semantic--acoustic alignment.
  12. Stage 3: Unfreezing the diffusion backbone with LoRA.
  13. Stage 4: Preference optimization with VRPO.
  14. Supervised Fine-Tuning of DIFFA-2
  15. Variance-Reduced Preference Optimization
  16. ...and 32 more sections

Figures (6)

  • Figure 1: Overview of DIFFA-2, including the dual-adapter architecture, multi-stage training pipeline (Stages 1–4), and iterative diffusion-based inference for general audio understanding.
  • Figure A.1: Examples of ASR Prompts in Stage 1
  • Figure A.2: Prompts of Audio QA Data Creation in Stage 2
  • Figure A.3: Examples of prompts in Stage 2
  • Figure A.4: Preference Data Generation Prompt
  • ...and 1 more figures