Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video Finetuning Improves Reasoning Between Frames

Ruiqi Yang, Tian Yun, Zihan Wang, Ellie Pavlick

TL;DR

This paper addresses the gap in temporal understanding when extending LVLMs from images to videos by comparing image-only and video-finetuned models under controlled conditions. It introduces Visual Chain-of-Thought (vCoT), which inserts two-step between-frame reasoning infills between consecutive frames, demonstrated on EgoSchema and evaluated with and without vCoT. Key findings show that vCoT markedly improves image-only models on long-form video QA, while video-finetuned models gain little from vCoT, implying they already capture frame transitions implicitly; moreover, video finetuning improves relational reasoning and transfers to static image tasks, such as i-Raven. The results highlight that explicit temporal prompts can aid frame-agnostic models, and that video finetuning imparts robust temporal biases with practical implications for cross-domain reasoning across static and dynamic visual tasks.

Abstract

Multimodal large language models (LLMs) have made rapid progress in visual understanding, yet their extension from images to videos often reduces to a naive concatenation of frame tokens. In this work, we investigate what video finetuning brings to multimodal LLMs. We propose Visual Chain-of-Thought (vCoT), an explicit reasoning process that generates transitional event descriptions between consecutive frames. Using vCoT, we systematically compare image-only LVLMs with their video-finetuned counterparts, both with and without access to these transitional cues. Our experiments show that vCoT significantly improves the performance of image-only models on long-form video question answering, while yielding only marginal gains for video-finetuned models. This suggests that the latter already capture frame-to-frame transitions implicitly. Moreover, we find that video models transfer this temporal reasoning ability to purely static settings, outperforming image models' baselines on relational visual reasoning tasks.

Video Finetuning Improves Reasoning Between Frames

TL;DR

This paper addresses the gap in temporal understanding when extending LVLMs from images to videos by comparing image-only and video-finetuned models under controlled conditions. It introduces Visual Chain-of-Thought (vCoT), which inserts two-step between-frame reasoning infills between consecutive frames, demonstrated on EgoSchema and evaluated with and without vCoT. Key findings show that vCoT markedly improves image-only models on long-form video QA, while video-finetuned models gain little from vCoT, implying they already capture frame transitions implicitly; moreover, video finetuning improves relational reasoning and transfers to static image tasks, such as i-Raven. The results highlight that explicit temporal prompts can aid frame-agnostic models, and that video finetuning imparts robust temporal biases with practical implications for cross-domain reasoning across static and dynamic visual tasks.

Abstract

Multimodal large language models (LLMs) have made rapid progress in visual understanding, yet their extension from images to videos often reduces to a naive concatenation of frame tokens. In this work, we investigate what video finetuning brings to multimodal LLMs. We propose Visual Chain-of-Thought (vCoT), an explicit reasoning process that generates transitional event descriptions between consecutive frames. Using vCoT, we systematically compare image-only LVLMs with their video-finetuned counterparts, both with and without access to these transitional cues. Our experiments show that vCoT significantly improves the performance of image-only models on long-form video question answering, while yielding only marginal gains for video-finetuned models. This suggests that the latter already capture frame-to-frame transitions implicitly. Moreover, we find that video models transfer this temporal reasoning ability to purely static settings, outperforming image models' baselines on relational visual reasoning tasks.

Paper Structure

This paper contains 16 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Visual Chain-of-Thought (vCoT) Infills
  4. Step 1: Common Visual Attributes.
  5. Step 2: Bridging Event Inference.
  6. Results
  7. Controlled experiment on EgoSchema
  8. Modality‑shuffling ablations
  9. Transferability to Relational Visual Reasoning
  10. Conclusion
  11. Acknowledgement
  12. Appendix
  13. Modality shuffling demonstration
  14. Frame captions as text infills
  15. LoRA fine‑tuning to neutralise data‑scale effects.
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Video LLMs typically sample a number of frames from a video and digest them as a sequence of concatenated visual tokens zhang2024llavanextvideo.
  • Figure 2: Visual CoT prompting pipeline. For each clip, transitional text infills are generated between every adjacent frame pair, interleaved with corresponding frames, and followed by the downstream question.
  • Figure 3: Visual CoT for inter-frame reasoning. Step 1: The model identifies shared visual attributes between two frames. Step 2: It infers a plausible intermediate event, which is then rephrased into a textual infill bridging the logical gap.
  • Figure 4: Shuffling to create conflicting modalities. Grey = retained modality; green = shuffled modality. Top row = text shuffle; bottom row = visual shuffle.