Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level

Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng Tian, Ajmal Saeed Mian, Mohit Bansal, Chen Chen

TL;DR

The paper defines Motion-Grounded Video Reasoning, a task where models must answer motion-related questions by generating pixel-level spatiotemporal masks. It introduces GroundMoRe, a large-scale dataset (1,715 clips, 249K masks) with four question types (Causal, Sequential, Counterfactual, Descriptive) to evaluate implicit reasoning and fine-grained grounding over time. The proposed MoRA model combines multimodal reasoning (LLMs), pixel-level grounding (SAM), and temporal localization to produce accurate spatiotemporal masks, achieving state-of-the-art results on GroundMoRe and outperforming prior baselines by a substantial margin. This work advances motion understanding in videos by coupling reasoning with concrete visual outputs, enabling more interpretable and temporally aware motion perception with broad potential applications.

Abstract

In this paper, we introduce Motion-Grounded Video Reasoning, a new motion understanding task that requires generating visual answers (video segmentation masks) according to the input question, and hence needs implicit spatiotemporal reasoning and grounding. This task extends existing spatiotemporal grounding work focusing on explicit action/motion grounding, to a more general format by enabling implicit reasoning via questions. To facilitate the development of the new task, we collect a large-scale dataset called GROUNDMORE, which comprises 1,715 video clips, 249K object masks that are deliberately designed with 4 question types (Causal, Sequential, Counterfactual, and Descriptive) for benchmarking deep and comprehensive motion reasoning abilities. GROUNDMORE uniquely requires models to generate visual answers, providing a more concrete and visually interpretable response than plain texts. It evaluates models on both spatiotemporal grounding and reasoning, fostering to address complex challenges in motion-related video reasoning, temporal perception, and pixel-level understanding. Furthermore, we introduce a novel baseline model named Motion-Grounded Video Reasoning Assistant (MORA). MORA incorporates the multimodal reasoning ability from the Multimodal LLM, the pixel-level perception capability from the grounding model (SAM), and the temporal perception ability from a lightweight localization head. MORA achieves respectable performance on GROUNDMORE outperforming the best existing visual grounding baseline model by an average of 21.5% relatively. We hope this novel and challenging task will pave the way for future advancements in robust and general motion understanding via video reasoning segmentation

Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level

TL;DR

The paper defines Motion-Grounded Video Reasoning, a task where models must answer motion-related questions by generating pixel-level spatiotemporal masks. It introduces GroundMoRe, a large-scale dataset (1,715 clips, 249K masks) with four question types (Causal, Sequential, Counterfactual, Descriptive) to evaluate implicit reasoning and fine-grained grounding over time. The proposed MoRA model combines multimodal reasoning (LLMs), pixel-level grounding (SAM), and temporal localization to produce accurate spatiotemporal masks, achieving state-of-the-art results on GroundMoRe and outperforming prior baselines by a substantial margin. This work advances motion understanding in videos by coupling reasoning with concrete visual outputs, enabling more interpretable and temporally aware motion perception with broad potential applications.

Abstract

In this paper, we introduce Motion-Grounded Video Reasoning, a new motion understanding task that requires generating visual answers (video segmentation masks) according to the input question, and hence needs implicit spatiotemporal reasoning and grounding. This task extends existing spatiotemporal grounding work focusing on explicit action/motion grounding, to a more general format by enabling implicit reasoning via questions. To facilitate the development of the new task, we collect a large-scale dataset called GROUNDMORE, which comprises 1,715 video clips, 249K object masks that are deliberately designed with 4 question types (Causal, Sequential, Counterfactual, and Descriptive) for benchmarking deep and comprehensive motion reasoning abilities. GROUNDMORE uniquely requires models to generate visual answers, providing a more concrete and visually interpretable response than plain texts. It evaluates models on both spatiotemporal grounding and reasoning, fostering to address complex challenges in motion-related video reasoning, temporal perception, and pixel-level understanding. Furthermore, we introduce a novel baseline model named Motion-Grounded Video Reasoning Assistant (MORA). MORA incorporates the multimodal reasoning ability from the Multimodal LLM, the pixel-level perception capability from the grounding model (SAM), and the temporal perception ability from a lightweight localization head. MORA achieves respectable performance on GROUNDMORE outperforming the best existing visual grounding baseline model by an average of 21.5% relatively. We hope this novel and challenging task will pave the way for future advancements in robust and general motion understanding via video reasoning segmentation

Paper Structure

This paper contains 11 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Benchmark: GroundMoRe
  4. Motion-Grounded Video Reasoning
  5. Video Collection
  6. Annotation Pipeline
  7. Experiments
  8. Baseline Models for Evaluation
  9. Our Method: MoRA
  10. Evaluation and Analysis
  11. Conclusion

Figures (3)

  • Figure 1: The illustration of the comparison between our Motion-Grounded Video Reasoning and previous video motion understanding tasks. Existing video motion understanding tasks (a)-(d) could at most address one or two key problems, either lacking fine-grained spatiotemporal perception or ignoring motion-related reasoning. (e) Our Motion-Grounded Video Reasoning considers both subject and object in motion as well as temporally adjacent events, performing challenging reasoning given four types of questions (Causal, Sequential, Counterfactual, and Descriptive) carefully designed in our GroundMoRe dataset and output spatiotemporal masks to indicate the answer visually at the pixel level. For instance, in the question "who needs to be passed or else the man in grey cannot easily score?", the motion "pass" and the subject "the man in grey" as well as an adjacent event "easily score" are provided in this question, the model needs reason about the object "the man in pink shorts", while output spatiotemporal masks (only between 0 to 32s where the motion "pass" happens). Such a paradigm fully grasps the spatiotemporal contexts of motion and provides an explainable response to evaluate the motion understanding ability. The colors of the questions are corresponded to the spatiotemporal masks.
  • Figure 2: Visualizations of GroundMoRe, including videos, questions, and visual answers (masks). Answer colors correspond to the masks. More examples can be found in Supplementary.
  • Figure 3: MoRA adopts the spatiotemporal pooling strategy and inserts the extra special [SEG] token. To enable the temporal localization ability, MoRA utilizes the extra [LOC] token to learn a binary temporal mask, which refines the direct SAM outputs.