Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning

TL;DR

This work tackles modal imbalance in audio-visual zero-shot learning by identifying quality and content discrepancies that cause over-reliance on the dominant modality. It introduces a discrepancy-aware attention network (DAAN) with two key components: quality discrepancy mitigation attention (QDMA), which yields sparse cross-modal attention and enhanced temporal information via a Temporal Convolutional Network, and contrastive sample-level gradient modulation (CSGM), which dynamically adjusts gradients at the sample level using a contribution rate derived from optimization and convergence rates. By jointly training with reconstructors and decoders and optimizing a composite loss that includes triplet, reconstruction, and cross-modal alignment terms, DAAN achieves state-of-the-art results on VGGSound, UCF101, and ActivityNet for both ZSL and GZSL, with ablations confirming the effectiveness of QDMA and CSGM. The proposed approach improves cross-modal fusion and generalization to unseen classes, offering a practical pathway to robust audio-visual recognition in settings with diverse and missing modalities.

Abstract

Audio-visual Zero-Shot Learning (ZSL) has attracted significant attention for its ability to identify unseen classes and perform well in video classification tasks. However, modal imbalance in (G)ZSL leads to over-reliance on the optimal modality, reducing discriminative capabilities for unseen classes. Some studies have attempted to address this issue by modifying parameter gradients, but two challenges still remain: (a) Quality discrepancies, where modalities offer differing quantities and qualities of information for the same concept. (b) Content discrepancies, where sample contributions within a modality vary significantly. To address these challenges, we propose a Discrepancy-Aware Attention Network (DAAN) for Enhanced Audio-Visual ZSL. Our approach introduces a Quality-Discrepancy Mitigation Attention (QDMA) unit to minimize redundant information in the high-quality modality and a Contrastive Sample-level Gradient Modulation (CSGM) block to adjust gradient magnitudes and balance content discrepancies. We quantify modality contributions by integrating optimization and convergence rate for more precise gradient modulation in CSGM. Experiments demonstrates DAAN achieves state-of-the-art performance on benchmark datasets, with ablation studies validating the effectiveness of individual modules.

Figures (3)

• Figure 1: Quality & Content Discrepancies in Audio-Visual Dataset: (a) Quality discrepancies exist between audio and visual modalities. The visual one holds more target-related data, causing the model's prediction to depend more on it. (b) Content discrepancies occur in samples. For the same category like playing basketball, distinct audio-visual samples result in diverse recognition outcomes because of different information biases (Sample 1 emphasizes the player, while Sample 2 emphasizes the ball).
• Figure 2: The DAAN architecture incorporates audio, visual, and textual features as inputs, simultaneously extracting semantic information. The QDMA unit removes redundant information from high-quality modalities to address quality discrepancies. Additionally, it extracts temporal embeddings of audio-visual features to enhance temporal information. The cross-attention layer facilitates information interaction between audio-visual features. The CSGM block dynamically adjusts the parameter gradients of various modules within the QDMA unit at the sample level, effectively eliminating content discrepancies among samples. The loss functions are illustrated in the lower-left area.
• Figure 3: Component Performance