Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

Faegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton

TL;DR

CoLeaF tackles weakly supervised AVVP by addressing the negative impact of cross-modal information on unaligned audible-only and visible-only events. The method uses two training branches, Reference and Anchor, with event-aware contrastive learning and cross-class knowledge distillation to transfer relational structure without increasing inference cost. It also introduces new evaluation metrics for audible-only and visible-only events, addressing a key evaluation gap, and demonstrates improvements on LLP and UnAV-100. Overall, CoLeaF delivers state-of-the-art performance and demonstrates strong generalization, with potential extensions to language-model–based cues.

Abstract

Weakly supervised audio-visual video parsing (AVVP) methods aim to detect audible-only, visible-only, and audible-visible events using only video-level labels. Existing approaches tackle this by leveraging unimodal and cross-modal contexts. However, we argue that while cross-modal learning is beneficial for detecting audible-visible events, in the weakly supervised scenario, it negatively impacts unaligned audible or visible events by introducing irrelevant modality information. In this paper, we propose CoLeaF, a novel learning framework that optimizes the integration of cross-modal context in the embedding space such that the network explicitly learns to combine cross-modal information for audible-visible events while filtering them out for unaligned events. Additionally, as videos often involve complex class relationships, modelling them improves performance. However, this introduces extra computational costs into the network. Our framework is designed to leverage cross-class relationships during training without incurring additional computations at inference. Furthermore, we propose new metrics to better evaluate a method's capabilities in performing AVVP. Our extensive experiments demonstrate that CoLeaF significantly improves the state-of-the-art results by an average of 1.9% and 2.4% F-score on the LLP and UnAV-100 datasets, respectively.

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

TL;DR

CoLeaF tackles weakly supervised AVVP by addressing the negative impact of cross-modal information on unaligned audible-only and visible-only events. The method uses two training branches, Reference and Anchor, with event-aware contrastive learning and cross-class knowledge distillation to transfer relational structure without increasing inference cost. It also introduces new evaluation metrics for audible-only and visible-only events, addressing a key evaluation gap, and demonstrates improvements on LLP and UnAV-100. Overall, CoLeaF delivers state-of-the-art performance and demonstrates strong generalization, with potential extensions to language-model–based cues.

Abstract

Weakly supervised audio-visual video parsing (AVVP) methods aim to detect audible-only, visible-only, and audible-visible events using only video-level labels. Existing approaches tackle this by leveraging unimodal and cross-modal contexts. However, we argue that while cross-modal learning is beneficial for detecting audible-visible events, in the weakly supervised scenario, it negatively impacts unaligned audible or visible events by introducing irrelevant modality information. In this paper, we propose CoLeaF, a novel learning framework that optimizes the integration of cross-modal context in the embedding space such that the network explicitly learns to combine cross-modal information for audible-visible events while filtering them out for unaligned events. Additionally, as videos often involve complex class relationships, modelling them improves performance. However, this introduces extra computational costs into the network. Our framework is designed to leverage cross-class relationships during training without incurring additional computations at inference. Furthermore, we propose new metrics to better evaluate a method's capabilities in performing AVVP. Our extensive experiments demonstrate that CoLeaF significantly improves the state-of-the-art results by an average of 1.9% and 2.4% F-score on the LLP and UnAV-100 datasets, respectively.
Paper Structure (11 sections, 12 equations, 5 figures, 6 tables)

This paper contains 11 sections, 12 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Preliminaries
  5. CoLeaF: Contrastive-Collaborative Learning Framework
  6. New Metrics for AVVP
  7. Experimental Results
  8. Ablation Studies
  9. State-of-the-Art Comparison
  10. Generalization on the Large-Scale UnAV-100 Dataset
  11. Conclusion

Figures (5)

  • Figure 1: An example of weakly supervised AVVP task. The method learns to temporally detect audible-only, visible-only, and audible-visible events in a video, using only video-level labels during training.
  • Figure 2: Performance of CMPAE gao2023collecting when using only unimodal (U) information versus exploiting both unimodal and cross-modal contexts (U+C) in detecting audible-only (A), visible-only (V) and audible-visible (AV) event types, in terms of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) rates.
  • Figure 3: The overall scheme of CoLeaF for the weakly supervised AVVP task. CoLeaF has two network branches, Reference and Anchor. The Reference branch focuses on unimodal information and cross-class relationships, while the Anchor branch exploits both unimodal and cross-modal contexts. The branches are trained simultaneously through the conventional video-level losses $\{\mathcal{{L}}^{Ref}_{video},\mathcal{{L}}^{Anchr}_{video}\}$ and our novel contrastive and knowledge distillation losses $\{\mathcal{L}^{Anch}_{Evt}, \mathcal{L}^{Ref}_{SelfMo},\mathcal{L}^{Anchr}_{CoCls}\}$. In inference, Anchor is deployed for the AVVP task.
  • Figure 4: Qualitative comparison to previous AVVP approaches (JoMoLD cheng2022joint and CMPAE gao2023collecting) training with weak labels.
  • Figure 5: Weakly supervised DAVE results on the UnAV-100 dataset. All methods use pre-trained VGGish hershey2017cnn and two-stream I3D carreira2017quo to generate the input tokens. The best and the second-best results are in Bold and underlined, respectively.