Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Visual-Language Models Effective in Action Recognition? A Comparative Study

Mahmoud Ali, Di Yang, François Brémond

TL;DR

This paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks.

Abstract

Current vision-language foundation models, such as CLIP, have recently shown significant improvement in performance across various downstream tasks. However, whether such foundation models significantly improve more complex fine-grained action recognition tasks is still an open question. To answer this question and better find out the future research direction on human behavior analysis in-the-wild, this paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks. Extensive experiments are conducted on recent fine-grained, human-centric action recognition datasets (e.g., Toyota Smarthome, Penn Action, UAV-Human, TSU, Charades) including action classification and segmentation.

Are Visual-Language Models Effective in Action Recognition? A Comparative Study

TL;DR

This paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks.

Abstract

Current vision-language foundation models, such as CLIP, have recently shown significant improvement in performance across various downstream tasks. However, whether such foundation models significantly improve more complex fine-grained action recognition tasks is still an open question. To answer this question and better find out the future research direction on human behavior analysis in-the-wild, this paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks. Extensive experiments are conducted on recent fine-grained, human-centric action recognition datasets (e.g., Toyota Smarthome, Penn Action, UAV-Human, TSU, Charades) including action classification and segmentation.

Paper Structure

This paper contains 12 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. SoTA Multi-modal Video Foundation Models
  3. Current Challenges on Action Recognition
  4. Zero-shot Action Classification
  5. Frame-wise Action Segmentation in Untrimmed Videos
  6. Evaluation Datasets
  7. Experimental Analysis and Discussion
  8. Comparisons on Zero-shot Action Classification
  9. Comparisons on Action Segmentation
  10. More Study
  11. Discussions and Novel Direction
  12. Conclusion

Figures (3)

  • Figure 1: Statistics of the results on different datasets.
  • Figure 2: TSNE visualization for Penn Action and Smarthome (CS) datasets.
  • Figure 3: Comparisons of text information using raw action labels, augmented action labels (Aug.) and full action description (Des.) on NTU and Smarthome.