Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crab$^{+}$: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Dongnuan Cai, Henghui Du, Chang Zhou, Xi Chen, Dan Guo, Hongyuan Zhang, Xuelong Li, Di Hu

Abstract

Developing Audio-Visual Large Language Models (AV-LLMs) for unified scene understanding is pivotal in multimodal intelligence. While instruction tuning enables pre-trained models with multi-task abilities, we observe that conventional multi-task unification methods often suffer from severe negative transfer, where nearly 55% of tasks degrade compared to single-task training. We attribute this phenomenon to audio-visual task heterogeneity, characterized by disparate task granularity and divergent capability demands, which lead to negative interference under joint training. To tackle this, we present Crab$^{+}$, a scalable and unified audio-visual scene understanding model that addresses task heterogeneity through explicit cooperation from both data and model perspectives. On the data side, we introduce AV-UIE v2, a comprehensive Audio-Visual Unified Instruction-tuning dataset with Explicit reasoning processes. It contains approximately 222K samples spanning 17 datasets and 7 tasks, enabling the model to capture cross-task relationships at different levels of granularity. On the model side, we design a unified interface to align heterogeneous task formulations, and propose Interaction-aware LoRA (I-LoRA), which explicitly models inter-task relationships via dynamic routing to coordinate distinct audio-visual interaction patterns, mitigating parameter interference. Extensive experiments show Crab$^{+}$ covers broader tasks than existing unified models while outperforming specialized models on various benchmarks. We successfully reverse the negative transfer trend, achieving positive transfer where multi-task learning surpasses single-task baselines in nearly 88% of tasks. These results hold across diverse AV-LLM paradigms and are validated through in-depth visualization, positioning Crab$^{+}$ as a robust step towards holistic audio-visual scene understanding.

Crab$^{+}$: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Abstract

Developing Audio-Visual Large Language Models (AV-LLMs) for unified scene understanding is pivotal in multimodal intelligence. While instruction tuning enables pre-trained models with multi-task abilities, we observe that conventional multi-task unification methods often suffer from severe negative transfer, where nearly 55% of tasks degrade compared to single-task training. We attribute this phenomenon to audio-visual task heterogeneity, characterized by disparate task granularity and divergent capability demands, which lead to negative interference under joint training. To tackle this, we present Crab, a scalable and unified audio-visual scene understanding model that addresses task heterogeneity through explicit cooperation from both data and model perspectives. On the data side, we introduce AV-UIE v2, a comprehensive Audio-Visual Unified Instruction-tuning dataset with Explicit reasoning processes. It contains approximately 222K samples spanning 17 datasets and 7 tasks, enabling the model to capture cross-task relationships at different levels of granularity. On the model side, we design a unified interface to align heterogeneous task formulations, and propose Interaction-aware LoRA (I-LoRA), which explicitly models inter-task relationships via dynamic routing to coordinate distinct audio-visual interaction patterns, mitigating parameter interference. Extensive experiments show Crab covers broader tasks than existing unified models while outperforming specialized models on various benchmarks. We successfully reverse the negative transfer trend, achieving positive transfer where multi-task learning surpasses single-task baselines in nearly 88% of tasks. These results hold across diverse AV-LLM paradigms and are validated through in-depth visualization, positioning Crab as a robust step towards holistic audio-visual scene understanding.
Paper Structure (17 sections, 7 equations, 12 figures, 7 tables)

This paper contains 17 sections, 7 equations, 12 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Audio-Visual Scene Understanding
  4. Unified Learning Paradigm
  5. The AV-UIE v2 Dataset
  6. Dataset Construction
  7. Dataset Analysis
  8. Method
  9. Unified Input-output Interface
  10. Architectural Instantiations
  11. Interaction-aware LoRA
  12. Experiments
  13. Tasks, Datasets and Evaluation Metrics
  14. Training Strategy
  15. Implementation Details
  16. ...and 2 more sections

Figures (12)

  • Figure 1: Multi-task vs. single-task LoRA on customized LLaMA2 and Qwen2.5-Omni. Green and red cells highlight tasks with performance gains and drops, respectively, under multi-task learning.
  • Figure 2: Overview of Crab$^{+}$. (a) Scalable and unified architecture capable of addressing diverse audio-visual scene understanding tasks. (b) Explicit cooperation that captures complex cross-task dependencies to improve overall performance.
  • Figure 3: Overview of AV-UIE v2 dataset. (a) Construction pipeline. We leverage MLLMs to augment raw data and simple labels with explicit reasoning processes. (b) Statistical distribution. The dataset integrates a comprehensive collection of source data from 17 datasets, covering 7 distinct scene understanding tasks.
  • Figure 4: Our unified input-output interface comprises a visual branch, an audio branch, a LLM backbone, and a segmentation module, trained on the AV-UIE v2 dataset with interaction-aware LoRA.
  • Figure 5: Interaction-aware LoRA comprises a shared matrix $A$ and multiple $B$ matrices, where inputs are dynamically routed to specific $B$ matrices.
  • ...and 7 more figures