Learning Musical Representations for Music Performance Question Answering

TL;DR

This work targets Audio-Visual Question Answering in dense music performances, where traditional AVQA methods struggle to exploit instrument-specific rhythm and source information and to align multimedia signals over time. Amuse introduces a multimodal interactive encoder plus rhythm and music-source encoders, along with a musical RoI highway, and trains a time-aware representation that aligns audio-visual content with rhythmic and source cues. The approach achieves state-of-the-art results on Music AVQA v1 and v2, with comprehensive ablations showing the critical role of early multimodal fusion, temporal musical representations, and RoI guidance. This framework advances practical music understanding in audiovisual contexts and motivates future self-supervised and domain-specific enhancements for broader multimodal reasoning tasks.

Abstract

Music performances are representative scenarios for audio-visual modeling. Unlike common scenarios with sparse audio, music performances continuously involve dense audio signals throughout. While existing multimodal learning methods on the audio-video QA demonstrate impressive capabilities in general scenarios, they are incapable of dealing with fundamental problems within the music performances: they underexplore the interaction between the multimodal signals in performance and fail to consider the distinctive characteristics of instruments and music. Therefore, existing methods tend to answer questions regarding musical performances inaccurately. To bridge the above research gaps, (i) given the intricate multimodal interconnectivity inherent to music data, our primary backbone is designed to incorporate multimodal interactions within the context of music; (ii) to enable the model to learn music characteristics, we annotate and release rhythmic and music sources in the current music datasets; (iii) for time-aware audio-visual modeling, we align the model's music predictions with the temporal dimension. Our experiments show state-of-the-art effects on the Music AVQA datasets. Our code is available at https://github.com/xid32/Amuse.

Figures (5)

• Figure 1: Sparse audio in general videos vs. continuous audio in music performance videos: The left natural video has sparse audio signals (occasional chirping) 9053174. The right is a sample from the Music AVQA, showing the dense and continuous audio signals of music performances li2022learning.
• Figure 2: Amuse framework integrates (a) and (b) combining multimodal interactive and musical-specialized representations for answering: (a) multimodal interactive encoder's audio, vision, and question modules are interconnected via adapters that perform cross-modal attention. (b) Source/Rhythm encoders extract and encode musical-specific characteristics such as rhythm and sound sources. Universal Encoder incorporates pretrained vision and audio source/rhythm encoders as depicted in Fig.\ref{['fig:predictor']}. A musical RoI extractor (light weight Yolo) detects music-related elements like instruments and performers. They are aligned with the temporal dimension.
• Figure 3: Visual frames and audio spectrograms are processed by rhythm and source encoders—one rhythm encoder and one source encoder for each modality (vision and audio)—to align their characteristics along the temporal dimension.
• Figure 4: Demonstration of audio-visual temporal and counting QA. We show examples that our model correctly handles audio-visual temporal and counting questions, while SoTA models LAVisH and DG-SCT fail.
• Figure 5: Distribution of importance scores for different modules (x-axis) in Amuse across various question types (y-axis). Each score ranges from 0 to 1, indicating the encoder significance as calculated by the attention layer for each question type in Music AVQA dataset. Higher scores reflect greater contributions to the final answers. Our results are averaged over three separate runs.