Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos

Arun Reddy, Ketul Shah, Corban Rivera, William Paul, Celso M. De Melo, Rama Chellappa

TL;DR

This work utilizes various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only and finds that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.

Abstract

In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.

An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos

TL;DR

This work utilizes various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only and finds that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.

Abstract

In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.
Paper Structure (5 sections, 2 figures, 2 tables)

This paper contains 5 sections, 2 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. EXPERIMENTS
  3. KNN Classification
  4. Zero-Shot Text-Based Classification
  5. DISCUSSION

Figures (2)

  • Figure 1: Examples of a real (left) and synthetic (right) video from RoCoG-v2. The datset consists of 7 gesture categories.
  • Figure 2: t-SNE plots for real and synthetic data. Real data is more meaningfully clustered compared to synthetic data, as the features are extracted using a ViT-B/16 model pre-trained on K710 and fine-tuned on K710 and K400. For (a), we use all real data whereas for (b), we use 50 samples per class chosen at random from the synthetic dataset.