Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos

TL;DR

This work demonstrates that a domain-specific foundation model for human action recognition can be effectively trained on a large corpus of unlabeled public videos from TikTok using VideoMAE V2, with ViT backbones, achieving state-of-the-art results on multiple benchmarks. The approach emphasizes data quality and cultural relevance, showing that carefully curated, unlabeled video collections can rival or surpass models trained on larger, traditional datasets. Key findings include pronounced gains from moderate pre-training data volumes with diminishing returns at scale, and that a smaller, high-quality TikTok subset can approach the performance achieved with standard datasets like UCF101 when fine-tuned. The study highlights practical implications for efficient, scalable pre-training of action-recognition models and underscores ethical considerations in sourcing public video data for foundation-model development.

Abstract

The increasing variety and quantity of tagged multimedia content on a variety of online platforms offer a unique opportunity to advance the field of human action recognition. In this study, we utilize 283,582 unique, unlabeled TikTok video clips, categorized into 386 hashtags, to train a domain-specific foundation model for action recognition. We employ VideoMAE V2, an advanced model integrating Masked Autoencoders (MAE) with Vision Transformers (ViT), pre-trained on this diverse collection of unstructured videos. Our model, fine-tuned on established action recognition benchmarks such as UCF101 and HMDB51, achieves state-of-the-art results: 99.05% on UCF101, 86.08% on HMDB51, 85.51% on Kinetics-400, and 74.27% on Something-Something V2 using the ViT-giant backbone. These results highlight the potential of using unstructured and unlabeled videos as a valuable source of diverse and dynamic content for training foundation models. Our investigation confirms that while initial increases in pre-training data volume significantly enhance model performance, the gains diminish as the dataset size continues to expand. Our findings emphasize two critical axioms in self-supervised learning for computer vision: (1) additional pre-training data can yield diminishing benefits for some datasets and (2) quality is more important than quantity in self-supervised learning, especially when building foundation models.

Figures (7)

• Figure 1: The hashtag selection and modification process.
• Figure 2: The clip extraction process from raw videos mined on TikTok.
• Figure 3: 100 Epoch Finetune: Final top-1 and top-5 accuracy by amount of pre-training data. We demonstrate the impact of the volume of pre-trained TikTok dataset on the fine-tuning accuracy for both Top-1 and Top-5 metrics.
• Figure 4: We compare the top-1 and top-5 accuracy of models pre-trained on TikTok actions, pre-trained from scratch, and pre-trained on UCF101 when fine-tuned to UCF101.
• Figure 5: Example clips from the TikTokActions dataset.