How Effective are Self-Supervised Models for Contact Identification in Videos

TL;DR

This study investigates how CNN-based self-supervised video models perform on detecting physical contact in videos, using eight SSL architectures pre-trained on Kinetics-400 with a uniform $R(2+1)D-18$ backbone. It evaluates these models on two diverse datasets, SSv2 and EK-100, under full fine-tuning and linear evaluation, plus template-based assessments for contact scenarios. The results show no single model consistently outperforms others across all settings; VideoMoCo and TCLR emerge as strong performers in different regimes, while EK-100 remains more challenging due to its ego-centric viewpoint and the difficulty of noun/action recognition. The findings highlight dataset-dependent strengths and limitations of SSL representations for contact identification and suggest future work in hand-detection integration and ViT-based architectures to enhance generalization across environments.

Abstract

The exploration of video content via Self-Supervised Learning (SSL) models has unveiled a dynamic field of study, emphasizing both the complex challenges and unique opportunities inherent in this area. Despite the growing body of research, the ability of SSL models to detect physical contacts in videos remains largely unexplored, particularly the effectiveness of methods such as downstream supervision with linear probing or full fine-tuning. This work aims to bridge this gap by employing eight different convolutional neural networks (CNNs) based video SSL models to identify instances of physical contact within video sequences specifically. The Something-Something v2 (SSv2) and Epic-Kitchen (EK-100) datasets were chosen for evaluating these approaches due to the promising results on UCF101 and HMDB51, coupled with their limited prior assessment on SSv2 and EK-100. Additionally, these datasets feature diverse environments and scenarios, essential for testing the robustness and accuracy of video-based models. This approach not only examines the effectiveness of each model in recognizing physical contacts but also explores the performance in the action recognition downstream task. By doing so, valuable insights into the adaptability of SSL models in interpreting complex, dynamic visual information are contributed.

Figures (6)

• Figure 1: lift lid off rice cooker. verb:- lift off, noun:- lid
• Figure 2: stir egg in pan using spatula, verb:- stir -in, noun:- egg
• Figure 3: SSv2 True category example
• Figure 4: SSv2 False category example
• Figure 5: Template-based evaluation for SSv2 dataset