UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing

TL;DR

This work tackles weakly-supervised audio-visual video parsing (AVVP), where only video-level labels are available, by introducing UWAV, a framework that learns from temporally coherent pseudo-labels. It builds a two-stage pipeline: (i) pre-trains transformer-based pseudo-label generators on a large supervised AV dataset to capture inter-segment dynamics, and (ii) uses uncertainty-aware supervision, uncertainty-weighted feature mixup, and class-balanced loss to train the AVVP inference module. The approach yields state-of-the-art results on LLP and AVE across multiple metrics, while also producing more accurate pseudo-labels than prior methods. Overall, UWAV reduces annotation costs and enhances generalization in weakly-supervised AVVP through explicit modeling of temporal structure and prediction uncertainty.

Abstract

Audio-Visual Video Parsing (AVVP) entails the challenging task of localizing both uni-modal events (i.e., those occurring exclusively in either the visual or acoustic modality of a video) and multi-modal events (i.e., those occurring in both modalities concurrently). Moreover, the prohibitive cost of annotating training data with the class labels of all these events, along with their start and end times, imposes constraints on the scalability of AVVP techniques unless they can be trained in a weakly-supervised setting, where only modality-agnostic, video-level labels are available in the training data. To this end, recently proposed approaches seek to generate segment-level pseudo-labels to better guide model training. However, the absence of inter-segment dependencies when generating these pseudo-labels and the general bias towards predicting labels that are absent in a segment limit their performance. This work proposes a novel approach towards overcoming these weaknesses called Uncertainty-weighted Weakly-supervised Audio-visual Video Parsing (UWAV). Additionally, our innovative approach factors in the uncertainty associated with these estimated pseudo-labels and incorporates a feature mixup based training regularization for improved training. Empirical results show that UWAV outperforms state-of-the-art methods for the AVVP task on multiple metrics, across two different datasets, attesting to its effectiveness and generalizability.

Figures (8)

• Figure 1: A weakly-supervised AVVP task example. Events, considered in this task, might be unimodal or multimodal. Even multimodal events, may not be temporally aligned in the audio and visual modalities, e.g. the cello might only be visible in the first few seconds but might produce music, throughout the video.
• Figure 2: UWAV framework: In stage 1, pseudo-label generation modules are equipped with the ability to capture temporal relationships between segments by pre-training on a large-scale, supervised, audio-visual event localization dataset. In stage 2, temporally coherent, uncertainty-weighted pseudo-labels, derived from the pre-trained pseudo-label generation module, are used to guide the learning of the inference model (HAN) aided by a class-balanced loss re-weighting and uncertainty-weighted feature mixup strategy. Note that we use the feature mixup strategy in both modalities while we only show the breakdown of the mixup operation for the audio modality.
• Figure 3: Comparison between predictions by UWAV and competing AVVP methods on the LLP dataset. "GT": ground truth.
• Figure A4: Sensitivity of $\alpha$ in the uncertainty-weighted feature mixup on the LLP dataset.
• Figure A6: Comparison between predictions by UWAV and competing AVVP methods on the LLP dataset. "GT": ground truth.