Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

TL;DR

Crab addresses the challenge of unified audio-visual scene understanding by enforcing explicit cross-task cooperation through an auxiliary AV-UIE dataset and a modular interaction-aware LoRA. The model uses a unified audio-visual interface (visual, audio, segmentation) feeding a large language model, with a router-guided multi-head LoRA ensuring distinct data-interaction capabilities are learned without destructive interference. Training occurs in two stages: feature alignment and instruction-tuning on AV-UIE, with explicit reasoning prompts enriching task relationships. Empirical results across AVE, AVVP, MUSIC-AVQA, AVS, and Ref-AVS demonstrate state-of-the-art or competitive performance, with ablations confirming the benefits of explicit reasoning and dynamic LoRA routing for cross-task transfer and specialization.

Abstract

In recent years, numerous tasks have been proposed to encourage model to develop specified capability in understanding audio-visual scene, primarily categorized into temporal localization, spatial localization, spatio-temporal reasoning, and pixel-level understanding. Instead, human possesses a unified understanding ability for diversified tasks. Therefore, designing an audio-visual model with general capability to unify these tasks is of great value. However, simply joint training for all tasks can lead to interference due to the heterogeneity of audiovisual data and complex relationship among tasks. We argue that this problem can be solved through explicit cooperation among tasks. To achieve this goal, we propose a unified learning method which achieves explicit inter-task cooperation from both the perspectives of data and model thoroughly. Specifically, considering the labels of existing datasets are simple words, we carefully refine these datasets and construct an Audio-Visual Unified Instruction-tuning dataset with Explicit reasoning process (AV-UIE), which clarifies the cooperative relationship among tasks. Subsequently, to facilitate concrete cooperation in learning stage, an interaction-aware LoRA structure with multiple LoRA heads is designed to learn different aspects of audiovisual data interaction. By unifying the explicit cooperation across the data and model aspect, our method not only surpasses existing unified audio-visual model on multiple tasks, but also outperforms most specialized models for certain tasks. Furthermore, we also visualize the process of explicit cooperation and surprisingly find that each LoRA head has certain audio-visual understanding ability. Code and dataset: https://github.com/GeWu-Lab/Crab

Figures (19)

• Figure 1: We present Crab, a unified audio-visual scene understanding model with explicit cooperation, which can complete various audio-visual tasks. It is trained on an instruction-tuning dataset with explicit reasoning process, which clarifies the cooperative relationship among tasks. Furthermore, to alleviate the interference caused by the learning process of complex audiovisual data and facilitate concrete cooperation, an interaction-aware LoRA structure is designed to enable the model focus on different aspects of data interaction.
• Figure 2: Our proposed AV-UIE dataset. (a) explains the specific process of dataset construction, and (b) is the data analysis for all tasks.
• Figure 3: The architecture of our unified audio-visual scene understanding model. It mainly consists of two parts: unified audio-visual interface, which consists of three multimodal branches, and a large language model with interaction-aware LoRA structure. The audio branch and visual branch process audio and video inputs respectively, while the segmentation branch is responsible for outputting the segmentation mask. The model is trained on our AV-UIE dataset, which clarifies the cooperation relationship among tasks, as marked by different colors on the right side of the figure. Content of same color in different tasks can help model establish cooperative relationship among tasks. Furthermore, to alleviate the interference caused by the learning process of complex audiovisual data, we design an interaction-aware LoRA structure to facilitate concrete cooperation.
• Figure 4: The architecture of interaction-aware LoRA structure. It comprises a shared matrix $A$ and multiple matrices $B$, also known as LoRA head. Matrix $A$ learns common multimodal representations, while each LoRA head is dedicated to learn certain audiovisual data interaction aspect.
• Figure 5: We visualize the router weights of three LoRA heads on different tasks. Figure (a) compares head-B1 and head-B2, while figure (b) compares head-B2 and head-B3. Different colors distinguish between tasks. The larger the router weight value, the greater the task's dependence on that LoRA head, indicating this LoRA head has a stronger ability to solve this type of task.