Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diving Deep into the Motion Representation of Video-Text Models

Chinmaya Devaraj, Cornelia Fermuller, Yiannis Aloimonos

TL;DR

This work interrogates whether video-text models truly understand motion in actions by introducing GPT-4 generated fine-grained motion descriptions and a motion-centric benchmark across Kinetics-400, UCF-101, and HMDB-51. It demonstrates that existing video-text models underperform human experts on motion-description retrieval and proposes a method that incorporates motion captions as classifier weights within a CLIP-based framework to improve motion understanding, achieving notable gains over baselines in zero-shot settings. The study highlights the importance of high-quality, motion-focused captions for training and evaluation, while offering a principled, theoretically motivated approach for transferring motion knowledge to unseen classes. Overall, the paper provides a benchmark, a practical method, and empirical evidence that better motion descriptions can enhance video-language understanding, with implications for motion-aware retrieval and recognition in real-world applications.

Abstract

Videos are more informative than images because they capture the dynamics of the scene. By representing motion in videos, we can capture dynamic activities. In this work, we introduce GPT-4 generated motion descriptions that capture fine-grained motion descriptions of activities and apply them to three action datasets. We evaluated several video-text models on the task of retrieval of motion descriptions. We found that they fall far behind human expert performance on two action datasets, raising the question of whether video-text models understand motion in videos. To address it, we introduce a method of improving motion understanding in video-text models by utilizing motion descriptions. This method proves to be effective on two action datasets for the motion description retrieval task. The results draw attention to the need for quality captions involving fine-grained motion information in existing datasets and demonstrate the effectiveness of the proposed pipeline in understanding fine-grained motion during video-text retrieval.

Diving Deep into the Motion Representation of Video-Text Models

TL;DR

This work interrogates whether video-text models truly understand motion in actions by introducing GPT-4 generated fine-grained motion descriptions and a motion-centric benchmark across Kinetics-400, UCF-101, and HMDB-51. It demonstrates that existing video-text models underperform human experts on motion-description retrieval and proposes a method that incorporates motion captions as classifier weights within a CLIP-based framework to improve motion understanding, achieving notable gains over baselines in zero-shot settings. The study highlights the importance of high-quality, motion-focused captions for training and evaluation, while offering a principled, theoretically motivated approach for transferring motion knowledge to unseen classes. Overall, the paper provides a benchmark, a practical method, and empirical evidence that better motion descriptions can enhance video-language understanding, with implications for motion-aware retrieval and recognition in real-world applications.

Abstract

Videos are more informative than images because they capture the dynamics of the scene. By representing motion in videos, we can capture dynamic activities. In this work, we introduce GPT-4 generated motion descriptions that capture fine-grained motion descriptions of activities and apply them to three action datasets. We evaluated several video-text models on the task of retrieval of motion descriptions. We found that they fall far behind human expert performance on two action datasets, raising the question of whether video-text models understand motion in videos. To address it, we introduce a method of improving motion understanding in video-text models by utilizing motion descriptions. This method proves to be effective on two action datasets for the motion description retrieval task. The results draw attention to the need for quality captions involving fine-grained motion information in existing datasets and demonstrate the effectiveness of the proposed pipeline in understanding fine-grained motion during video-text retrieval.
Paper Structure (31 sections, 1 equation, 3 figures, 8 tables)

This paper contains 31 sections, 1 equation, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Benchmark
  4. Proposed method
  5. Problem setting
  6. Architecture setting
  7. Theoretical justification
  8. Results
  9. Conclusion
  10. Limitations
  11. Ethical Considerations
  12. Acknowledgements
  13. Implementation details of our method
  14. Problem Setting:
  15. Training
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Example captions from ActivityNet, MSR-VTT, and our own GPT-4 generated fine-grained motion description for Kinetics-400 classes. Our generated motion descriptions solely describe the motion of the action, whereas other datasets typically use verbs to describe the scene.
  • Figure 2: Schematic representation of the generation of motion descriptions in existing action datasets.
  • Figure 3: Schematic representation of our approach encoding motion description in video-text model pipeline: We integrate motion information as classifier weight in a supervised training paradigm. We finetune the image encoder to integrate the motion information while classifying videos in the kinetics400 dataset.