Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding

TL;DR

This work tackles the inefficiency of current thinking-with-frames approaches for long video understanding by introducing SpecTemp, a dual-model speculative temporal reasoning framework that decouples temporal perception from reasoning. A lightweight draft MLLM rapidly proposes salient frames within predicted temporal regions, while a stronger target MLLM performs high-level temporal reasoning and verification, iterating until convergence. Training is supported by SpecTemp-80K, a dual-level annotated dataset built with GPT-4o, and optimized via supervised fine-tuning followed by reinforcement learning with tailored rewards for each model. Empirical results across eight benchmarks show SpecTemp achieves competitive accuracy with notably reduced inference latency, highlighting its potential for efficient real-time long-video reasoning.

Abstract

Long video understanding is essential for human-like intelligence, enabling coherent perception and reasoning over extended temporal contexts. While the emerging thinking-with-frames paradigm, which alternates between global temporal reasoning and local frame examination, has advanced the reasoning capabilities of video multi-modal large language models (MLLMs), it suffers from a significant efficiency bottleneck due to the progressively growing and redundant multi-modal context. To address this, we propose SpecTemp, a reinforcement learning-based Speculative Temporal reasoning framework that decouples temporal perception from reasoning via a cooperative dual-model design. In SpecTemp, a lightweight draft MLLM rapidly explores and proposes salient frames from densely sampled temporal regions, while a powerful target MLLM focuses on temporal reasoning and verifies the draft's proposals, iteratively refining its attention until convergence. This design mirrors the collaborative pathways of the human brain, balancing efficiency with accuracy. To support training, we construct the SpecTemp-80K dataset, featuring synchronized dual-level annotations for coarse evidence spans and fine-grained frame-level evidence. Experiments across multiple video understanding benchmarks demonstrate that SpecTemp not only maintains competitive accuracy but also significantly accelerates inference compared with existing thinking-with-frames methods.

Figures (12)

• Figure 1: Illustration of SpecTemp's iterative speculation-verification process. Target MLLM predicts temporal regions (Clue Tokens); Draft MLLM proposes frames; Target MLLM verifies with History Tokens across iterative rounds.
• Figure 2: Overview of data generation pipeline and composition
• Figure 3: Inference latency breakdown. SpecTemp achieves the lowest latency (2.3s) by delegating dense sampling to the draft model.
• Figure 4: Visualization of SpecTemp's iterative speculation-verification process with target-draft model collaboration.
• Figure 5: SpecTemp-80K dataset statistics.