LocoMotion: Learning Motion-Focused Video-Language Representations

TL;DR

This paper proposes LocoMotion to learn from motion-focused captions that describe the movement and temporal progression of local object motions, and proposes verb-variation paraphrasing to increase the caption variety and learn the link between primitive motions and high-level verbs.

Abstract

This paper strives for motion-focused video-language representations. Existing methods to learn video-language representations use spatial-focused data, where identifying the objects and scene is often enough to distinguish the relevant caption. We instead propose LocoMotion to learn from motion-focused captions that describe the movement and temporal progression of local object motions. We achieve this by adding synthetic motions to videos and using the parameters of these motions to generate corresponding captions. Furthermore, we propose verb-variation paraphrasing to increase the caption variety and learn the link between primitive motions and high-level verbs. With this, we are able to learn a motion-focused video-language representation. Experiments demonstrate our approach is effective for a variety of downstream tasks, particularly when limited data is available for fine-tuning. Code is available: https://hazeldoughty.github.io/Papers/LocoMotion/

Figures (9)

• Figure 1: Motion-Focused Video-Language Representations. Video-text pairs from web data have a spatial focus. To learn motion-focused video-language representations, we create motion-focused video-text pairs for pre-training by adding synthetic motions to videos and automatically generating new captions describing these motions.
• Figure 2: Caption Content of video-language datasets. Left: We show the average number of nouns, adjectives, verbs, adverbs, and adpositions per captions for 5 popular pre-training (top) and downstream datasets (bottom). Right: The percentage of captions uniquely identifiable using only the nouns. Current video-language datasets have a spatial focus demonstrated by the average number of nouns each caption contains and the percentage of captions that can be uniquely identified by only their nouns.
• Figure 3: Captioning with VLMs also results in a spatial-focus. Examples generated with VideoChat2.
• Figure 4: LocoMotion learns a motion-focused video-language representation. Given a video, we randomly sample an object, a size, and a starting location giving the appearance of the initial frame. We generate local object motion by translating and rotating the object across time. As we know the motion parameters we can create corresponding text to accurately describe the object's motion in the video. Our verb-variation paraphrasing allows us to diversify the vocabulary and structure of this motion description and link low-level motion to more high-level verbs. Using the resulting video-text pairs to pre-train video, text and cross-modal encoders with masking, matching, and contrastive losses results in a motion-focused video-language representation.
• Figure 5: Motion-Focused Video-Text Pairs. Our motion generation adds clearer and more varied motion to input videos. With our motion description and verb-variation paraphrasing, we are able to automatically create diverse descriptions of these motions.