Label-anticipated Event Disentanglement for Audio-Visual Video Parsing

TL;DR

This paper tackles the decoding stage of Audio-Visual Video Parsing (AVVP) where multiple events may overlap in time. It introduces LEAP, a label semantic-based projection paradigm that projects latent audio-visual features into semantically independent label embeddings via a cross-modal Transformer, enabling explicit disentanglement of overlapping events. A semantic-aware optimization strategy complements LEAP with an $L_{avss}$ loss based on the audio-visual Intersection over Union ($EIoU$), aligning cross-modal feature similarity with semantic overlap. Empirical results on LLP show LEAP achieving state-of-the-art AVVP performance and improved generalization to AVEL, validating that semantically guided decoding enhances both interpretability and accuracy across modalities and tasks.

Abstract

Audio-Visual Video Parsing (AVVP) task aims to detect and temporally locate events within audio and visual modalities. Multiple events can overlap in the timeline, making identification challenging. While traditional methods usually focus on improving the early audio-visual encoders to embed more effective features, the decoding phase -- crucial for final event classification, often receives less attention. We aim to advance the decoding phase and improve its interpretability. Specifically, we introduce a new decoding paradigm, \underline{l}abel s\underline{e}m\underline{a}ntic-based \underline{p}rojection (LEAP), that employs labels texts of event categories, each bearing distinct and explicit semantics, for parsing potentially overlapping events.LEAP works by iteratively projecting encoded latent features of audio/visual segments onto semantically independent label embeddings. This process, enriched by modeling cross-modal (audio/visual-label) interactions, gradually disentangles event semantics within video segments to refine relevant label embeddings, guaranteeing a more discriminative and interpretable decoding process. To facilitate the LEAP paradigm, we propose a semantic-aware optimization strategy, which includes a novel audio-visual semantic similarity loss function. This function leverages the Intersection over Union of audio and visual events (EIoU) as a novel metric to calibrate audio-visual similarities at the feature level, accommodating the varied event densities across modalities. Extensive experiments demonstrate the superiority of our method, achieving new state-of-the-art performance for AVVP and also enhancing the relevant audio-visual event localization task.

Figures (6)

• Figure 1: Illustration of the AVVP task and different event decoding paradigms. (a) The AVVP task is required to parse audio events, visual events, and audio-visual events within the video. Each segment may contain multiple overlapping events. Given the latent audio/visual embedding, (b) the typical decoding paradigm 'MMIL' directly predicts multiple event classes by using simple linear layers. (c) We propose to elucidate the disentanglement of the potentially overlapping semantics through the projection of latent features into multiple, semantically separate label embeddings.
• Figure 2: Overview of our method. (a) Our network for audio-visual video parsing. Prior typical audio-visual encoders can be employed for earlier audio and visual feature embedding, such as HAN tian2020unified and MM-Pyr yu2021mm. We focus on enhancing the later decoder with the proposed label semantic-based projection (LEAP) strategy. Specifically, we explicitly introduce the separate label embeddings of all event classes and then disentangle potentially overlapping events by projecting the audio or visual features into those label embeddings. (b) The illustration of LEAP. LEAP models the cross-modal relations between audio/visual with label embeddings. The label embeddings corresponding to the ground truth events are enhanced to be discriminative. The intermediate cross-attention matrix $\bm{A}^{lm}$ and the final enhanced label embedding $\bm{F}^{lm}$ is used for segment-level and video-level event predictions, respectively. (c) For effective projection and model optimization, we consider the supervision from uni-modal labels at both the video level and segment level ($\mathcal{L}_{basic}$). We also design a new audio-visual semantic similarity loss function $\mathcal{L}_{avss}$ to regularize the model by considering cross-modal relations at the feature level.
• Figure 3: Comparison between LEAP and typical MMIL in parsing audio and visual events in each class.$\bigtriangleup$ denotes the performance improvements of our method compared to MMIL and "Avg." denotes the average results of all the event classes. MM-Pyr yu2021mm is used as the audio-visual encoder and the event-level metrics are reported.
• Figure 4: Qualitative examples of audio-visual video parsing. Compared to MMIL, the proposed LEAP performs better in distinguishing the semantics of non-overlapping and overlapping events.
• Figure 5: More qualitative video examples of audio-visual video parsing. Best view in color and zoom in.