MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs

TL;DR

MotionSight tackles the bottleneck in fine-grained video motion understanding for MLLMs by introducing a training-free, object-focused visual spotlight and motion blur to separate object and camera motion cues. It couples this zero-shot prompting with MotionVid-QA, a large-scale hierarchical dataset built from diverse sources to enable SFT and DPO training, and demonstrates state-of-the-art open-source performance on motion benchmarks with strong generalization to broader video tasks. The work also shows that finetuning on MotionVid-QA yields MotionChat with competitive accuracy on FAVOR-Bench and enhances motion-aware video generation, while providing robust ablations and scalability analyses. Overall, MotionSight offers a practical, training-free boost to fine-grained motion understanding and a valuable, open data resource for the community.

Abstract

Despite advancements in Multimodal Large Language Models (MLLMs), their proficiency in fine-grained video motion understanding remains critically limited. They often lack inter-frame differencing and tend to average or ignore subtle visual cues. Furthermore, while visual prompting has shown potential in static images, its application to video's temporal complexities, particularly for fine-grained motion understanding, remains largely unexplored. We investigate whether inherent capability can be unlocked and boost MLLMs' motion perception and enable distinct visual signatures tailored to decouple object and camera motion cues. In this study, we introduce MotionSight, a novel zero-shot method pioneering object-centric visual spotlight and motion blur as visual prompts to effectively improve fine-grained motion understanding without training. To convert this into valuable data assets, we curated MotionVid-QA, the first large-scale dataset for fine-grained video motion understanding, with hierarchical annotations including SFT and preference data, Θ(40K) video clips and Θ(87K) QAs. Experiments show MotionSight achieves state-of-the-art open-source performance and competitiveness with commercial models. In particular, for fine-grained motion understanding we present a novel zero-shot technique and a large-scale, high-quality dataset. All the code and annotations will be publicly available.

Figures (19)

• Figure 1: Motivation and approach overview. (a) Temporal dynamics inherent in motion distinguish videos from static images. (b) Existing MLLMs show limitations in fine-grained motion detection, whereas our approach excels in accurately understanding object and camera motion. (c) Our approach shows superior performance on MotionBench and FAVOR-Bench compared to SOTA.
• Figure 2: Overview of the interaction process.Left: Our $\mathtt{MotionSight}$ pipeline captions high-quality data, transforming it into data assets. Right: This data undergoes rigorous filtering to align with human preferences, resulting in our high-quality dataset $\mathtt{MotionVid-QA}$.
• Figure 3: Comparison of our method with other existing methods. Directly applying image visual prompts can lead to misinterpretation. By employing decoupled object-guided motion focusing and inter-frame information enhancement, our method addresses the challenge faced by previous methods.
• Figure 3: Quantitative results on FAVOR-Bench. We selected representative MLLMs as baselines for comparison. We computed the OM (object motion) metric by averaging all metrics excluding the CM (camera motion) metric in FAVOR-Bench.
• Figure 4: The detailed pipeline of $\mathtt{MotionSight}$. Our method includes query-based motion decoupling, gating based on object motion and camera motion. Subsequently, it selectively passes through modules based on the decoupled type. Then, we carefully designed a template prompt for MLLMs to understand our enhanced input and make final decisions.