Efficient Motion-Aware Video MLLM

TL;DR

This work introduces EMA, an Efficient Motion-Aware video MLLM that processes compressed-domain GOP inputs to unify spatial and motion information into compact tokens. By decoupling and fusing I-frame features with motion vectors through a motion-aware GOP encoder, EMA reduces input redundancy while enhancing motion representation, enabling strong video understanding with lower inference costs. A new MotionBench benchmark assesses four motion types (linear, curved, rotational, and contact-based) to specifically evaluate motion understanding. Empirical results show EMA achieving state-of-the-art or competitive performance on MotionBench and standard video QA benchmarks, with notable efficiency gains and good scalability to long videos. Together, EMA and MotionBench offer a practical, scalable path toward more motion-aware, token-efficient video MLLMs in real-world applications.

Abstract

Most current video MLLMs rely on uniform frame sampling and image-level encoders, resulting in inefficient data processing and limited motion awareness. To address these challenges, we introduce EMA, an Efficient Motion-Aware video MLLM that utilizes compressed video structures as inputs. We propose a motion-aware GOP (Group of Pictures) encoder that fuses spatial and motion information within a GOP unit in the compressed video stream, generating compact, informative visual tokens. By integrating fewer but denser RGB frames with more but sparser motion vectors in this native slow-fast input architecture, our approach reduces redundancy and enhances motion representation. Additionally, we introduce MotionBench, a benchmark for evaluating motion understanding across four motion types: linear, curved, rotational, and contact-based. Experimental results show that EMA achieves state-of-the-art performance on both MotionBench and popular video question answering benchmarks, while reducing inference costs. Moreover, EMA demonstrates strong scalability, as evidenced by its competitive performance on long video understanding benchmarks.

Figures (4)

• Figure 1: Comparison of sampling from decoded frames versus the GOPs (Group Of Pictures) from compressed video stream. Compressed video encoding generates tokens at only 1/T the length of sampled frames for the same clip, while capturing motion information more directly. It also shows greater efficiency on Video-QA (average of MSVD-QA and MSRVTT-QA xu2017video) and MotionBench within our EMA framework, achieving higher accuracy with less inference time (red arrow shows inference speed up).
• Figure 2: Description of the H.264 codec modes for a single GOP (Group of Pictures). P/B-frames are decoded sequentially in the decoding order. Decoding relies on motion vectors, which record the movement of each macroblock in the current frame relative to those in reference frames.
• Figure 3: An illustrative diagram of the overall model architecture. The compressed-domain video stream is divided into GOPs (Groups of Pictures), and each GOP is encoded using our designed GOP encoder. After concatenation, the encoded GOPs are input into the LLM along with text instructions. On the left side of the figure is the detailed structure of the GOP encoder, which decouples frame and motion encoding. It fuses the frame features with the aggregated motion feature sequence to produce a fixed-length GOP feature containing both spatial and motion information.
• Figure 4: Examples of data from MotionBench. We show four types of examples: linear, curved, rotation, and contact. Yellow arrows in the video frames indicate the motion trajectories of the same specified object. Different trajectory patterns correspond to different data types.