Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning

TL;DR

The paper introduces Chain-of-Frames (CoF), a frame-grounded chain-of-thought approach for video LLMs that directly references video frames within reasoning traces. It develops CoF-Data by leveraging real videos (VideoEspresso) and synthetic videos (Clevrer), enabling scalable training without auxiliary frame-selection modules. Fine-tuning open video LLMs on CoF-Data yields consistent performance gains across five benchmarks and reduces hallucinations, often surpassing state-of-the-art models. The method is simple, scalable, and enhances interpretability by linking reasoning steps to specific frames, offering a practical path to more reliable video understanding in LLMs.

Abstract

Recent work has shown that eliciting Large Language Models (LLMs) to generate reasoning traces in natural language before answering the user's request can significantly improve their performance across tasks. This approach has been extended to multimodal LLMs, where the models can produce chain-of-thoughts (CoT) about the content of input images and videos. In this work, we propose to obtain video LLMs whose reasoning steps are grounded in, and explicitly refer to, the relevant video frames. For this, we first create CoF-Data, a large dataset of diverse questions, answers, and corresponding frame-grounded reasoning traces about both natural and synthetic videos, spanning various topics and tasks. Then, we fine-tune existing video LLMs on this chain-of-frames (CoF) data. Our approach is simple and self-contained, and, unlike existing approaches for video CoT, does not require auxiliary networks to select or caption relevant frames. We show that our models based on CoF are able to generate chain-of-thoughts that accurately refer to the key frames to answer the given question. This, in turn, leads to improved performance across multiple video understanding benchmarks, for example, surpassing leading video LLMs on Video-MME, MVBench, and VSI-Bench, and notably reducing the hallucination rate. Code available at https://github.com/SaraGhazanfari/CoF}{github.com/SaraGhazanfari/CoF.

Figures (16)

• Figure 1: A CoF reasoning trace generated by our CoF-4B model.
• Figure 2: CoF models vs the baseline models.
• Figure 4: CoF-Data. We show examples of our training data with chain-of-frames reasoning, including video, question, answer and reasoning trace. We include samples from CoF-Datareal (real video, top row) and CoF-Datasynth (synthetic video, bottom), created as described in Sec. \ref{['sec:data']}.
• Figure 5: Overview of our two-step pipeline for generating CoF-Data. Step 1 adjusts the frame IDs while preserving frame-caption alignment. Step 2 utilizes raw annotations to generate CoF triplets (question, frame-aware reasoning trace, answer). For this, we leverage Llama3.1-8B a manual template with the synthetic videos from Clevrer.
• Figure 6: Distribution of frame references in the Chain-of-Frames training data. The left pie chart illustrates the distribution for CoF-Datareal, having fewer frames per reasoning trace, whereas CoF-DATA-synth demonstrates a more balanced frame distribution due to controlled synthetic video generation. The right pie chart shows the overall distribution for the CoF-Data.