Video-CoM: Interactive Video Reasoning via Chain of Manipulations

TL;DR

This work introduces Video-CoM, a paradigm for interactive video reasoning that treats video content as an active cognitive workspace. It models reasoning as a chain of manipulations (find-segment, find-frame, spatial-zoom) managed by Video-CoM and trained with a two-stage process (SFT and RA-GRPO) on Video-CoM-Instruct. A dedicated 18K QA dataset with dense spatio-temporal annotations supports manipulation-driven supervision, enabling step-level rewards that ground intermediate reasoning. Across nine benchmarks, Video-CoM achieves state-of-the-art or competitive results with significantly less training data, demonstrating the value of active visual interaction for grounded, multi-step video reasoning.

Abstract

Recent multimodal large language models (MLLMs) have advanced video understanding, yet most still "think about videos" ie once a video is encoded, reasoning unfolds entirely in text, treating visual input as a static context. This passive paradigm creates a semantic bottleneck: models cannot rewatch, refocus, or verify evidence, leading to shallow visual reasoning on tasks requiring fine grained spatio temporal understanding. In this work, we introduce Interactive Video Reasoning, a new paradigm that transforms video into an active cognitive workspace, enabling models to "think with videos". Our model, Video CoM, reasons through a Chain of Manipulations (CoM), performing iterative visual actions to gather and refine evidence. To support this behavior, we construct Video CoM Instruct, an 18K instruction tuning dataset curated for multi step manipulation reasoning. Beyond supervised learning, we further optimize the manipulation policy via reinforcement learning with reasoning aware Group Relative Policy Optimization (GRPO). Unlike prior work that relies solely on sparse answer rewards, our method introduces step level reasoning rewards, guiding the model toward grounded and consistent reasoning. Video CoM achieves strong results across nine video reasoning benchmarks, improving average performance by 3.6 percent over recent state of the art models, while training on only 25K SFT and 3K GRPO video samples, significantly fewer than comparable large scale models. Ablation studies demonstrate that reasoning aware rewards improve both accuracy and interpretability. Code: https://github.com/mbzuai-oryx/Video-CoM

Figures (10)

• Figure 1: Most existing video reasoning models think about videos rather than think with them. Once a video is encoded, reasoning unfolds purely in text, causing thinking tokens to drift toward world knowledge rather than visual evidence. Moreover, these models are typically trained with sparse accuracy rewards that supervise only the final answer, leading to inconsistencies between reasoning steps and predictions (top, Video-R1 feng2025videor1). We address these limitations with Interactive Video Reasoning, where our model (bottom, Video-CoM) reasons with videos through a coherent chain of manipulations, actively gathering and integrating visual evidence throughout reasoning.
• Figure 2: Overview of the Video-CoM-Instruct-18K dataset pipeline. The dataset is built through five stages: i) Curating videos for active reasoning, combining information-dense videos from existing datasets and manually curated clips; ii) Designing chain-of-manipulations, defining the atomic operations (find-segment, find-frame, spatial-zoom); iii) Interactive video reasoning data generation, where Gemini-2.5-Pro produces manipulation-targeted QA pairs; iv) Data verification using model agreement filtering ensuring quality; and v) Dense spatio-temporal annotation, which enables step-level reasoning rewards for GRPO. The resulting Video-CoM-Instruct dataset contains 18K QA pairs (15K for SFT and 3K for GRPO), supporting interactive, manipulation-based video reasoning.
• Figure 3: Overview of Interactive Video Reasoning and the proposed Reasoning-Aware GRPO (RA-GRPO) framework. The model, Video-CoM, reasons with videos through a Chain of Manipulations (CoM), iteratively applying visual operations to gather and refine visual evidence. Each manipulation produces new observations that are fed back into the model for continued reasoning. Because multiple manipulation trajectories can lead to the same correct answer, accuracy-only rewards cannot distinguish grounded from spurious reasoning paths. RA-GRPO resolves this by introducing step-level reasoning rewards that explicitly evaluate each manipulation. The examples illustrate how intermediate reasoning steps are verified, providing partial credit for correct intermediate actions and guiding the model toward consistent, visually grounded reasoning trajectories.
• Figure 4: Comparison of visual attention. Video-R1 feng2025videor1 (top) focuses primarily on text tokens, indicating reliance on world knowledge rather than visual evidence. Video-CoM (bottom) maintains dynamic attention to vision tokens across reasoning rounds, demonstrating active visual reasoning.
• Figure 5: Proportion of videos selected from existing sources for constructing Video-CoM-Instruct. The figure shows the final composition of the $9$K videos drawn from large-scale existing datasets used in the SFT stage of Video-CoM-Instruct, including ActivityNet caba2015activitynet, Charades sigurdsson2016hollywood_charades, VidOR shang2019annotating, YouCookII zhou2017youcookii, YouTube videos collected by LLaVA-178K zhang2024video, PerceptionTest patraucean2023perception, CLEVRER yi2019clevrer, and Ego4D grauman2022ego4d.