AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering

TL;DR

AV-Master tackles the challenge of complex audio-visual question answering by introducing a dual-path architecture that separately models temporal focus and modality preferences. The Temporal Dynamic Perception Path progressively concentrates on question-relevant audio-visual segments, while the Global Preference Activation Path decouples modality contributions to capture global cues. A dual-path optimization with a cross-path contrastive loss enforces consistency and complementarity between the two paths, yielding robust cross-modal representations. Experiments on MUSIC-AVQA, MUSIC-AVQA-R, MUSIC-AVQA-v2.0, and AVQA demonstrate state-of-the-art or competitive performance, with strong gains on complex reasoning tasks and resilience to data biases and varying input conditions.

Abstract

Audio-Visual Question Answering (AVQA) requires models to effectively utilize both visual and auditory modalities to answer complex and diverse questions about audio-visual scenes. However, existing methods lack sufficient flexibility and dynamic adaptability in temporal sampling and modality preference awareness, making it difficult to focus on key information based on the question. This limits their reasoning capability in complex scenarios. To address these challenges, we propose a novel framework named AV-Master. It enhances the model's ability to extract key information from complex audio-visual scenes with substantial redundant content by dynamically modeling both temporal and modality dimensions. In the temporal dimension, we introduce a dynamic adaptive focus sampling mechanism that progressively focuses on audio-visual segments most relevant to the question, effectively mitigating redundancy and segment fragmentation in traditional sampling methods. In the modality dimension, we propose a preference-aware strategy that models each modality's contribution independently, enabling selective activation of critical features. Furthermore, we introduce a dual-path contrastive loss to reinforce consistency and complementarity across temporal and modality dimensions, guiding the model to learn question-specific cross-modal collaborative representations. Experiments on four large-scale benchmarks show that AV-Master significantly outperforms existing methods, especially in complex reasoning tasks.

Figures (9)

• Figure 1: Illustration of the AVQA task and the comparison of our method with previous work. Our method employs dynamic adaptive focus sampling to capture key audio-visual segments and predicts modal preferences through global preference activation to enhance the model, ultimately generating the correct answer.
• Figure 2: Overview of AV-Master. We utilize three separate pre-trained encoders to extract features from video, audio, and question inputs. The encoded features are then fed into the temporal dynamic perception path and the global preference activation path, respectively. Finally, the model predicts the correct answer based on the outputs of these two paths.
• Figure 3: The pipeline of (a) audio-visual focus capture and (c) audio-visual key fusion in the temporal dynamic perception path, where (b) represents the specific implementation process of focus sampling in (a) audio-visual focus capture. SAB and CAB represent the self-attention block and the cross-attention block, respectively. represent the input predefined CLS tokens (serve as audio-visual templates), represent the audio-visual features at a certain time step, and is a learned bias.
• Figure 4: The ablation study on the lengths of visual and audio segments and comparison with QA-TIGER.
• Figure 5: The ablation study on input modalities and comparison with other popular models (AVTS and QA-TIGER).