Music's Multimodal Complexity in AVQA: Why We Need More than General Multimodal LLMs

TL;DR

Music AVQA exposes limitations of general multimodal models by requiring beat-accurate, instrument-specific reasoning over densely layered audio-visual content. The paper argues for specialized input pipelines, spatial-temporal architectures, and musical priors, supported by empirical analyses of datasets (MUSIC-AVQA, v2.0, and MUSIC-AVQA-R) and method families. It proposes concrete directions such as incorporating fine-grained musical event cues, mid-level musical structure, latent reasoning trajectories, and supervised chain-of-thought to enhance interpretability and generalization. Collectively, these contributions lay a foundation for robust, domain-aware multimodal understanding in music and suggest broader applicability to other complex, structured modalities.

Abstract

While recent Multimodal Large Language Models exhibit impressive capabilities for general multimodal tasks, specialized domains like music necessitate tailored approaches. Music Audio-Visual Question Answering (Music AVQA) particularly underscores this, presenting unique challenges with its continuous, densely layered audio-visual content, intricate temporal dynamics, and the critical need for domain-specific knowledge. Through a systematic analysis of Music AVQA datasets and methods, this position paper identifies that specialized input processing, architectures incorporating dedicated spatial-temporal designs, and music-specific modeling strategies are critical for success in this domain. Our study provides valuable insights for researchers by highlighting effective design patterns empirically linked to strong performance, proposing concrete future directions for incorporating musical priors, and aiming to establish a robust foundation for advancing multimodal musical understanding. This work is intended to inspire broader attention and further research, supported by a continuously updated anonymous GitHub repository of relevant papers: https://github.com/xid32/Survey4MusicAVQA.

Figures (5)

• Figure 1: Contrast between (i) conventional QA and (ii) Music-AVQA with dense audio. Panel (i) shows an isolated sound (barking) and synchronized action, which are relatively easy to detect. Panel (ii) exemplifies music's complexity, featuring overlapping instruments and rhythmic patterns. Such dense and continuous audio-visual signals demand fine-grained temporal and spatial reasoning through cross-modal comparisons and they are more challenging than conventional multimodal QA.
• Figure 2: Accuracy comparison of Music AVQA models across representative question types, grouped by modality: (a–b) Audio, (c–e) Visual, and (f–i) Audio-Visual. Each bar corresponds to a model and is color-coded based on whether it incorporates spatial-temporal design for the relevant task type: bars in HTML]d5e7d4green, HTML]e4d9eapurple, and HTML]ffe8d2orange represent models that apply spatial-temporal modeling to Audio-related, Visual-related, and Audio-Visual-related question answering, respectively; bars in HTML]d9e7fbblue represent models without spatial-temporal design. Across most categories, models with spatial-temporal components tend to perform more accurately, particularly on tasks requiring temporal reasoning or spatial localization. These patterns suggest that incorporating spatial-temporal design supports more effective reasoning in musically structured multimodal environments.
• Figure 3: Radar plots showing the per-type average accuracy of model groups with and without spatial-temporal design across 13 QA categories on (a) Music-AVQA 9879157 and (b) Music-AVQA-R ma2024look. Each axis corresponds to a QA type spanning audio, visual, and audio-visual reasoning, including the overall average (Total-Average). The filled HTML]d5e7d4green polygon in Figure \ref{['subfig:music-avqa-radar']} and HTML]e4d9eapurple polygon in Figure \ref{['subfig:music-avqa-r-radar']} represent the mean accuracy across QA types for models with spatial-temporal design, while the HTML]d9e7fbblue polygon represents the average performance of models without such design. Models with spatial-temporal design consistently achieve higher accuracy across all modality groups. These advantages persist under distribution shift in the robustness-focused Music-AVQA-R dataset.
• Figure 4: Representative examples for the four common music performance scene types. (a) Solo performance: a single musician highlights individual virtuosity on one instrument. (b) Ensemble of the same instrument: multiple players of identical (or closely related) instruments create timbral thickness and homogeneous harmony. (c) Ensemble of different instruments: a heterogeneous group blends distinct tonal colours and enables richer contrapuntal interaction. (d) Culture-specific ensemble: a traditional instrumental configuration (e.g. guzheng quartet, gamelan group) that captures the performance idioms of a particular musical culture.
• Figure 5: Examples of Music AVQA question types spanning audio, visual, and audio-visual modalities, including counting, comparison, localization, existential, and temporal QA.