Noise-Tolerant Learning for Audio-Visual Action Recognition

TL;DR

A noise-tolerant learning framework that significantly improves the robustness of the action recognition model and surpasses the baselines by a clear margin and establishes a benchmark of real-world noisy correspondence in audio-visual data by relabeling the Kinetics dataset.

Abstract

Recently, video recognition is emerging with the help of multi-modal learning, which focuses on integrating distinct modalities to improve the performance or robustness of the model. Although various multi-modal learning methods have been proposed and offer remarkable recognition results, almost all of these methods rely on high-quality manual annotations and assume that modalities among multi-modal data provide semantically relevant information. Unfortunately, the widely used video datasets are usually coarse-annotated or collected from the Internet. Thus, it inevitably contains a portion of noisy labels and noisy correspondence. To address this challenge, we use the audio-visual action recognition task as a proxy and propose a noise-tolerant learning framework to find anti-interference model parameters against both noisy labels and noisy correspondence. Specifically, our method consists of two phases that aim to rectify noise by the inherent correlation between modalities. First, a noise-tolerant contrastive training phase is performed to make the model immune to the possible noisy-labeled data. To alleviate the influence of noisy correspondence, we propose a cross-modal noise estimation component to adjust the consistency between different modalities. As the noisy correspondence existed at the instance level, we further propose a category-level contrastive loss to reduce its interference. Second, in the hybrid-supervised training phase, we calculate the distance metric among features to obtain corrected labels, which are used as complementary supervision to guide the training. Extensive experiments on a wide range of noisy levels demonstrate that our method significantly improves the robustness of the action recognition model and surpasses the baselines by a clear margin.

Figures (8)

• Figure 1: Cross-entropy loss $vs.$ epoch on UCF101 dataset under noise for clean-annotated and noisy label samples. (a) Training the late-fusion audio-visual network with 60% noisy labels and no noisy correspondence. (b) Training the late-fusion audio-visual network with both 60% noisy labels and 60% noisy correspondence. From the figure, we can see that the noisy correspondence will confuse the loss of noisy label samples and clean-annotated samples.
• Figure 2: The motivation of cross-modal noise estimation. In this figure, we denote $(\bm{v_i},\bm{a_i})$, $(\bm{v_j},\bm{a_j})$ and $(\bm{v_k},\bm{a_k})$ as the features of audio-visual pairs, and show their similarity in the common space. (a) Correct Correspondence: If $(\bm{v_i},\bm{a_i})$ has correct correspondence, we use the audio modality to find the cluster samples $\bm{a_j}$ and $\bm{a_k}$, and their corresponding visual modality $\bm{v_j}$ and $\bm{v_k}$ are also close to $\bm{v_i}$ in common space, vice versa. (b) Noisy Correspondence: If $\bm{a_i}$ is irrelevant to $\bm{v_i}$, the corresponding visual modality $\bm{v_j}$ and $\bm{v_k}$ are far away from $\bm{v_i}$ in common space.
• Figure 3: The framework of our proposed method. The noise-tolerant contrastive training phase and hybrid-supervised training phase proceed iteratively until converged. The modal-specific networks are shared parameters between these two phases.
• Figure 4: Histogram of instance counts over the entire 19,404 validation dataset, sorted by the percentage of noisy correspondence among each class. Here we show the top 20 classes with noisy labels and top 20 clean classes.
• Figure 5: Clip-level top-1 accuracy vs. epoch on noisy training dataset and clean test dataset of UCF101. The label noise ratios are set as: 20% asymmetric, 40% asymmetric, 60% symmetric and 80% symmetric.