Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning

TL;DR

Weaver tackles the challenges of video reasoning by moving beyond text-centric Chain-of-Thought to an end-to-end multimodal agentic framework. It introduces a toolkit of visual perception tools and an interleaved vision-language reasoning process, trained in two stages: supervised finetuning for reliable tool invocation and reinforcement learning for exploration of tool compositions. The approach is supported by two datasets, Weaver-SFT-10K and Weaver-RL-12K, enabling trajectory-rich training and trajectory-free RL data, respectively. Empirical results across diverse long-video benchmarks show consistent gains over baselines and existing CoT methods, highlighting improved perception, reduced hallucination, and better spatio-temporal reasoning. Collectively, Weaver advances scalable, tool-enabled, perceptual reasoning toward more capable multimodal AI systems in video understanding tasks.

Abstract

Video reasoning constitutes a comprehensive assessment of a model's capabilities, as it demands robust perceptual and interpretive skills, thereby serving as a means to explore the boundaries of model performance. While recent research has leveraged text-centric Chain-of-Thought reasoning to augment these capabilities, such approaches frequently suffer from representational mismatch and restricted by limited perceptual acuity. To address these limitations, we propose Weaver, a novel, end-to-end trainable multimodal reasoning agentic system. Weaver empowers its policy model to dynamically invoke diverse tools throughout the reasoning process, enabling progressive acquisition of crucial visual cues and construction of authentic multimodal reasoning trajectories. Furthermore, we integrate a reinforcement learning algorithm to allow the system to freely explore strategies for employing and combining these tools with trajectory-free data. Extensive experiments demonstrate that our system, Weaver, enhances performance on several complex video reasoning benchmarks, particularly those involving long videos.

Figures (9)

• Figure 1: Our method, Weaver, leverages an interleaved visual-text reasoning paradigm, enabling the flexible combination and invocation of tools to progressively acquire visual information and generate multimodal reasoning trajectories towards final answer. As shown in (c), in comparison to the baseline methods illustrated in (a) and (b), Weaver successfully utilizes both the frame selection and spatial grounding tools to obtain a precise highlighted bounding box for the counting problem, which demonstrates the superiority of our approach.
• Figure 2: Overview of Weaver agentic system. During the multi-turn interleaved reasoning process, Weaver concatenates all tokens generated in previous rounds as input for subsequent rounds, continuing this procedure until a final answer is obtained. Consequently, the entire reasoning process can be interpreted as a multi-round conversational exchange.
• Figure 3: Data Pipeline and Statistics. Panel (a) illustrates the data construction pipeline for Weaver-SFT and Weaver-RL, beginning with two textual CoT reasoning datasets and resulting in two high-quality reasoning datasets. Panel (b) presents the statistical analysis of Weaver-SFT. Panel (c) shows the specific compositions of two dataset, fire emojis mean the content will be supervised during training.
• Figure 4: Tool Usage analysis for Weaver in different evaluation benchmarks.
• Figure 5: Visualization result of Weaver. The red regions indicate the model responses, the blue regions denote the tool-calling processes, and the purple regions correspond to the newly inserted visual information.