TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning

TL;DR

This work addresses the need for temporally grounded visual reasoning by introducing TwiFF-2.7M, a large-scale dataset of dynamic VCoT derived from 2.7M video clips, and TwiFF-Bench, a rigorous benchmark that jointly evaluates reasoning plausibility and final answers in open-ended dynamic scenarios. It presents TwiFF, a unified model that interleaves future-frame generation with textual reasoning to produce temporally coherent visual cues, demonstrating significant gains over static VCoT and textual baselines on dynamic visual question answering tasks. Key findings show that integrating visual and textual modalities is crucial, that physically plausible visual cues improve accuracy, and that dynamic VCoT can compress information while retaining essential context. Together, TwiFF-2.7M and TwiFF-Bench provide foundational data and evaluation tools to advance dynamic visual reasoning and suggest future reinforcement learning directions to optimize chain-of-thought plausibility.

Abstract

Visual Chain-of-Thought (VCoT) has emerged as a promising paradigm for enhancing multimodal reasoning by integrating visual perception into intermediate reasoning steps. However, existing VCoT approaches are largely confined to static scenarios and struggle to capture the temporal dynamics essential for tasks such as instruction, prediction, and camera motion. To bridge this gap, we propose TwiFF-2.7M, the first large-scale, temporally grounded VCoT dataset derived from $2.7$ million video clips, explicitly designed for dynamic visual question and answer. Accompanying this, we introduce TwiFF-Bench, a high-quality evaluation benchmark of $1,078$ samples that assesses both the plausibility of reasoning trajectories and the correctness of final answers in open-ended dynamic settings. Building on these foundations, we propose the TwiFF model, a unified modal that synergistically leverages pre-trained video generation and image comprehension capabilities to produce temporally coherent visual reasoning cues-iteratively generating future action frames and textual reasoning. Extensive experiments demonstrate that TwiFF significantly outperforms existing VCoT methods and Textual Chain-of-Thought baselines on dynamic reasoning tasks, which fully validates the effectiveness for visual question answering in dynamic scenarios. Our code and data is available at https://github.com/LiuJunhua02/TwiFF.

Figures (15)

• Figure 1: Dynamic VCoT (TwiFF) vs. TCoT and Static VCoT: Blue denotes correct model outputs, and red denotes errors or factual inconsistencies.
• Figure 2: Data generation. We generate TwiFF-2.7M in three stages.
• Figure 3: TwiFF Overview. Both TwiFF-2.7M and TwiFF-Bench consist of dynamic VCoT with interleaved image-text reasoning, covering three task types: instructional, predictive, and camera.
• Figure 4: Topic distribution for TwiFF-2.7M.
• Figure 5: The impact of the alignment between VCoT reasoning and observed future events in the video on the answer score on TwiFF-Bench.