CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

TL;DR

CACE-Net addresses audio-visual event localization by introducing a bidirectional cross-modal framework that combines audio-visual co-guidance attention (AVCA), background-event contrast enhancement (BECE), and targeted modal feature fine-tuning. AVCA enables mutual guidance between visual and audio streams to suppress noise and align cross-modal cues, while BECE uses supervised contrastive learning to sharpen the distinction between event and background, aided by robust encoders fine-tuned on downstream data. The approach yields state-of-the-art results on the AVE dataset, with additional validation on UnAV-100 for generalization, and is supported by thorough ablations demonstrating the contribution of each component. The work offers a practical, robust strategy for multimodal event localization in unconstrained videos, with implications for improved scene understanding and cross-modal fusion in real-world applications $F_o = F_{av} + \lambda F_{FT}$ and related losses $\mathcal{L} = \mathcal{L}^c + \frac{1}{N}\sum_t (\mathcal{L}_t^e + \mathcal{L}_t^{sup}) + \mathcal{L}^{contrast}$.

Abstract

The audio-visual event localization task requires identifying concurrent visual and auditory events from unconstrained videos within a network model, locating them, and classifying their category. The efficient extraction and integration of audio and visual modal information have always been challenging in this field. In this paper, we introduce CACE-Net, which differs from most existing methods that solely use audio signals to guide visual information. We propose an audio-visual co-guidance attention mechanism that allows for adaptive bi-directional cross-modal attentional guidance between audio and visual information, thus reducing inconsistencies between modalities. Moreover, we have observed that existing methods have difficulty distinguishing between similar background and event and lack the fine-grained features for event classification. Consequently, we employ background-event contrast enhancement to increase the discrimination of fused feature and fine-tuned pre-trained model to extract more refined and discernible features from complex multimodal inputs. Specifically, we have enhanced the model's ability to discern subtle differences between event and background and improved the accuracy of event classification in our model. Experiments on the AVE dataset demonstrate that CACE-Net sets a new benchmark in the audio-visual event localization task, proving the effectiveness of our proposed methods in handling complex multimodal learning and event localization in unconstrained videos. Code is available at https://github.com/Brain-Cog-Lab/CACE-Net.

Figures (6)

• Figure 1: An example of audio-visual event localization task, where we can identify an event as a mandolin only if both the sound and visual content are mandolin in the video segment.
• Figure 2: Overview of our proposed network framework, which consists of three parts: audio-visual co-guidance, background-event contrastive learning, and pre-trained model targeted fine-tuning.
• Figure 3: Schematic diagram of audio-visual co-guidance attention. Based on visual and audio modalities providing different information for event localization, visual and audio acquire guidance signals from each other and adaptively conduct cross-modal attentional guidance.
• Figure 4: Prediction results of our method on two relatively difficult samples selected from the AVE dataset. The black font represents the background, the blue font represents the categories of events that the model predicted correctly, and the red is the incorrect predictions output by the model. GT stands for Ground Truth.
• Figure 5: Visualization of different attentional guidance methods. The top row is ground truth as the original image, the middle row is the CMGA method, and the last row is our method (CACE-Net).