VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool

TL;DR

This work tackles the challenge of building video-focused chain-of-thought data for video OpenQA by introducing an active-learning–driven automatic annotation tool. It creates three datasets—VideoCoT, TopicQA, and TopicCoT—paired with a simple video reasoning benchmark, and demonstrates that semi-automatic CoT generation coupled with expert refinement substantially improves open-ended, video-based reasoning in multimodal LLMs. The approach reduces labeling cost while enhancing reasoning quality, as evidenced by improved CoT fluency, coverage of scene and temporal dynamics, and stronger performance on OE QA, especially with hybrid training. This has practical implications for advancing video understanding in multimodal systems and opens avenues for scalable, explainable reasoning in video QA tasks.

Abstract

Multimodal large language models (MLLMs) are flourishing, but mainly focus on images with less attention than videos, especially in sub-fields such as prompt engineering, video chain-of-thought (CoT), and instruction tuning on videos. Therefore, we try to explore the collection of CoT datasets in videos to lead to video OpenQA and improve the reasoning ability of MLLMs. Unfortunately, making such video CoT datasets is not an easy task. Given that human annotation is too cumbersome and expensive, while machine-generated is not reliable due to the hallucination issue, we develop an automatic annotation tool that combines machine and human experts, under the active learning paradigm. Active learning is an interactive strategy between the model and human experts, in this way, the workload of human labeling can be reduced and the quality of the dataset can be guaranteed. With the help of the automatic annotation tool, we strive to contribute three datasets, namely VideoCoT, TopicQA, TopicCoT. Furthermore, we propose a simple but effective benchmark based on the collected datasets, which exploits CoT to maximize the complex reasoning capabilities of MLLMs. Extensive experiments demonstrate the effectiveness our solution.

Figures (8)

• Figure 1: The case analysis of video question answering.
• Figure 2: The process of automatic dataset construction for VideoCoT and TopicCoT.
• Figure 3: After multiple rounds of training, the quality score of the generated CoT is improved from 0.07 to 0.97.
• Figure 4: The topic and question distribution for VideoCoT and TopicCoT.
• Figure 5: The length distribution of our dataset, where the y-axis represents the number of samples whose length is the x-axis value.