Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering

TL;DR

Audio-Visual Question Answering requires coherent reasoning across audio, visual, and linguistic modalities. The proposed QSTar framework jointly guides audio-visual feature learning with the question, enhances cross-modal interactions through a Spatial--Temporal--Frequency module, and refines reasoning with a prompting-based Query Context Reasoning block. Empirical results on MUSIC-AVQA show state-of-the-art performance across most question types, with ablations confirming the contribution of each component. This approach improves instrument-level localization and interpretation in complex, polyphonic video scenes, offering a powerful paradigm for query-guided multimodal reasoning in music contexts.

Abstract

Audio--Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio--visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial--Temporal--Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio--visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on several AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code and pretrained models will be released after publication.

Figures (5)

• Figure 1: Illustration of AVQA task and comparison between prior methods and QSTar. (a) Input sample. (b) Prior works rely on object-level cues, struggling with subtle motions (e.g., inactive flute player). (c) QSTar enhances spatial--temporal--frequency interaction. Green patches highlight spatial focus. Red box shows diminishing high-frequency bands as the flute stops, while violin remains active.
• Figure 2: Overall framework of the proposed QSTar method. We use pre-trained encoders to extract audio, visual, and linguistic features. The Query-Guided Multimodal Correlation module (yellow area) refines $F_a$ and $F_v$ using query information, resulting in $F_{aq}'$ and $F_{vq}'$. These features are further enhanced by the Spatial--Temporal--Frequency Interaction module (purple area), which integrates Spatial--Temporal Interaction (STI) and Temporal--Frequency Interaction (TFI), using additional frequency-aware features from AST gong21b_ast. The Query Context Reasoning block (green area) incorporates prompt-based context ($F_\mathrm{prompt}$) to guide multimodal fusion for answer prediction. For brevity, we remove self-attention units.
• Figure 3: Qualitative results in the MUSIC-AVQA li2022musicavqa dataset. (a) compares answer predictions between QSTar and QA-TIGER kim2025tiger on two examples. (b) visualizes spatial, temporal, and frequency focuses from QSTar’s STFI module. Green boxes highlight the patch-level visual attention at key timestamps selected via audio-based temporal focus. A 2D spectrogram provides an overview of frequency dynamics for better interpretability.
• Figure 4: Prediction results of our method for different question types of Audio QA (counting, comparative) and Visual QA (counting, location) in MUSIC-AVQA li2022musicavqa, with video ids: "00007961", "00002646", "00002454", and "00004109", respectively. We provide the top 3 answers predicted by our method in sequence.
• Figure 5: Prediction results of our method for different question types of Audio-Visual QA (existential, counting, location, comparative, temporal) in MUSIC-AVQA li2022musicavqa, with video ids: "00003803", "00004995", "00008437", "00005464", and "00004026", respectively. We provide the top 3 answers predicted by our method in sequence.