Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality

TL;DR

This work addresses the vulnerability of Audio-Visual Question Answering (AVQA) systems to missing input modalities by introducing a two-component framework. The Relation-aware Missing Modal (RMM) generator recalls information about the missing modality using cross-modal cues, while the Audio-Visual Relation-aware (AVR) diffusion model enhances both real and pseudo features through cross-modal diffusion and denoising. The authors define the learning objective with a dedicated RMM recall loss and an AVE diffusion-enhancement loss, integrated into a total loss for robust AVQA performance. Experiments on MUSIC-AVQA and AVQA2022 demonstrate substantial improvements under missing modalities and show consistent gains across multiple AVQA backbones, highlighting the method's practical impact for real-world multi-modal reasoning tasks.

Abstract

Recent Audio-Visual Question Answering (AVQA) methods rely on complete visual and audio input to answer questions accurately. However, in real-world scenarios, issues such as device malfunctions and data transmission errors frequently result in missing audio or visual modality. In such cases, existing AVQA methods suffer significant performance degradation. In this paper, we propose a framework that ensures robust AVQA performance even when a modality is missing. First, we propose a Relation-aware Missing Modal (RMM) generator with Relation-aware Missing Modal Recalling (RMMR) loss to enhance the ability of the generator to recall missing modal information by understanding the relationships and context among the available modalities. Second, we design an Audio-Visual Relation-aware (AVR) diffusion model with Audio-Visual Enhancing (AVE) loss to further enhance audio-visual features by leveraging the relationships and shared cues between the audio-visual modalities. As a result, our method can provide accurate answers by effectively utilizing available information even when input modalities are missing. We believe our method holds potential applications not only in AVQA research but also in various multi-modal scenarios.

Figures (5)

• Figure 1: Concept diagram of our methodology. Leveraging mutual cues in trimodal relations to recall and enhance missing information.
• Figure 2: Overall architecture of the proposed AVQA framework for missing modality (audio missing example). We introduce (a) Relation-aware Missing Modal (RMM) generator and (b) Audio-Visual Relation-aware (AVR) diffusion model. More details for learning RMM generator and AVR diffusion are in Sec. 3.1 and Sec. 3.2.
• Figure 3: RMM generator operates in two scenarios: (a) generating pseudo audio when audio input is missing, and (b) generating pseudo visuals when visual input is missing. Based on the addressing vector (i.e.,${a}^{vt}_i$, ${a}^{at}_i$), pseudo modality feature is obtained by a weight summation with the corresponding generator. Each generator in the three modalities shares weights.
• Figure 4: AVR diffusion process illustrates how the model learns to generate enhanced features for both audio and visual modalities by leveraging cross-modal knowledge. (a) depicts the diffusion and reverse process of concatenated features between real features, while (b) represents the reverse process of features where the pseudo feature and the real feature are concatenated.
• Figure 5: AVQA results on the MUSIC-AVQA dataset vary based on (a) the missing ratio of visual and audio modalities, and (b) the varying missing ratios for visual (upper) or audio (lower) modalities.