ActNetFormer: Transformer-ResNet Hybrid Method for Semi-Supervised Action Recognition in Videos

TL;DR

ActNetFormer tackles semi-supervised video action recognition by marrying cross-architecture pseudo-labeling with cross-architecture contrastive learning. It deploys a dual-model setup consisting of a primary 3D-ResNet50 and an auxiliary ViT-S, processing video clips at differing frame rates to capture complementary spatial and temporal cues. Pseudo-labels generated across architectures guide learning on unlabeled data, while a video-level contrastive objective aligns and enriches the joint representations. The approach achieves state-of-the-art performance on Kinetics-400 and UCF-101 with limited labeled data, and ablations reveal the effectiveness of the cross-architecture design and the contrastive component for robust action understanding in real-world scenarios.

Abstract

Human action or activity recognition in videos is a fundamental task in computer vision with applications in surveillance and monitoring, self-driving cars, sports analytics, human-robot interaction and many more. Traditional supervised methods require large annotated datasets for training, which are expensive and time-consuming to acquire. This work proposes a novel approach using Cross-Architecture Pseudo-Labeling with contrastive learning for semi-supervised action recognition. Our framework leverages both labeled and unlabelled data to robustly learn action representations in videos, combining pseudo-labeling with contrastive learning for effective learning from both types of samples. We introduce a novel cross-architecture approach where 3D Convolutional Neural Networks (3D CNNs) and video transformers (VIT) are utilised to capture different aspects of action representations; hence we call it ActNetFormer. The 3D CNNs excel at capturing spatial features and local dependencies in the temporal domain, while VIT excels at capturing long-range dependencies across frames. By integrating these complementary architectures within the ActNetFormer framework, our approach can effectively capture both local and global contextual information of an action. This comprehensive representation learning enables the model to achieve better performance in semi-supervised action recognition tasks by leveraging the strengths of each of these architectures. Experimental results on standard action recognition datasets demonstrate that our approach performs better than the existing methods, achieving state-of-the-art performance with only a fraction of labeled data. The official website of this work is available at: https://github.com/rana2149/ActNetFormer.

Figures (3)

• Figure 1: Comparison of performance between different architectural models.
• Figure 2: Architecture of the proposed framework.
• Figure 3: Analysis of different hyperparameters which includes Threshold $\tau$, Batch ratio $B_u$, Loss weight $\gamma$, Loss weight $\beta$.