Semi-supervised Active Learning for Video Action Detection

TL;DR

The paper tackles label-efficient video action detection by unifying semi-supervised learning and active learning. It introduces NoiseAug for data-driven sample informativeness in AL and fft-attention (FFT-based high-pass filtering) to improve SSL pseudo-label quality by focusing on action regions, all within a Mean Teacher SSL framework. Empirical results on UCF101-24 and JHMDB-21 show state-of-the-art performance among semi-supervised and weakly supervised methods, with notable gains when combining AL and SSL; generalization to YouTube-VOS demonstrates applicability to other dense video tasks. The approach is simple to implement and readily extensible to other dense prediction problems in video, offering practical benefits for label-scarce scenarios.

Abstract

In this work, we focus on label efficient learning for video action detection. We develop a novel semi-supervised active learning approach which utilizes both labeled as well as unlabeled data along with informative sample selection for action detection. Video action detection requires spatio-temporal localization along with classification, which poses several challenges for both active learning informative sample selection as well as semi-supervised learning pseudo label generation. First, we propose NoiseAug, a simple augmentation strategy which effectively selects informative samples for video action detection. Next, we propose fft-attention, a novel technique based on high-pass filtering which enables effective utilization of pseudo label for SSL in video action detection by emphasizing on relevant activity region within a video. We evaluate the proposed approach on three different benchmark datasets, UCF-101-24, JHMDB-21, and Youtube-VOS. First, we demonstrate its effectiveness on video action detection where the proposed approach outperforms prior works in semi-supervised and weakly-supervised learning along with several baseline approaches in both UCF101-24 and JHMDB-21. Next, we also show its effectiveness on Youtube-VOS for video object segmentation demonstrating its generalization capability for other dense prediction tasks in videos. The code and models is publicly available at: \url{https://github.com/AKASH2907/semi-sup-active-learning}.

Figures (4)

• Figure 1: Overview of our proposed approach: During the training phase, we take the labeled and unlabeled data at equal ratio to train the model together. We apply strong and weak augmentations to all input samples. All detection output is passed through our FFT filter to get a weight mask, which is used to compute the final consistency loss between teacher and student model output. During the active video selection phase, we take trained student model and pass $R$ variants with NoiseAug and score the sample. We select top K videos for further annotation and cycle back to the next training phase.
• Figure 2: Analysis on selection criteria: We compare our proposed AL selection with other selection baselines using the same SSL training setup on UCF101-24 (a-b) and JHMDB-21 (c-d).
• Figure 3: Qualitative analysis of FFT based high pass filter: We show the input frames (first row) and corresponding weights (bottom row) using proposed FFT filter. The FFT method gives higher weight towards the edges of detected action regions while suppressing background. Red: low weight, blue: medium weight, green: high weight
• Figure 4: FFT Analysis: Left: FFT filter on teacher(T)-student(S) for SSL training. Right: Effect of radius (r) on FFT filter. Both are on UCF101-24 for 10% labeled data.