Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?

TL;DR

Video-Holmes introduces a Sherlock Holmes-inspired benchmark to assess complex, multi-clue video reasoning in multimodal LLMs, addressing a gap where existing benchmarks emphasize perception over integration. It includes 270 suspense shorts and 1,837 questions across seven reasoning tasks, demanding active clue seeking and cross-segment reasoning, with DeepSeek-generated QA explanations. Evaluations show SOTA MLLMs struggle with integrating clues, with most accuracies below 40% and the best at 45%, though RL thinking prompts and audio cues offer improvements. The work provides in-depth reasoning-process analyses and demonstrates that high-quality human-annotated cues drastically boost performance, underscoring the need for advances in true multimodal reasoning capabilities and establishing Video-Holmes as a challenging, widely applicable benchmark for future research.

Abstract

Recent advances in CoT reasoning and RL post-training have been reported to enhance video reasoning capabilities of MLLMs. This progress naturally raises a question: can these models perform complex video reasoning in a manner comparable to human experts? However, existing video benchmarks primarily evaluate visual perception and grounding abilities, with questions that can be answered based on explicit prompts or isolated visual cues. Such benchmarks do not fully capture the intricacies of real-world reasoning, where humans must actively search for, integrate, and analyze multiple clues before reaching a conclusion. To address this issue, we present Video-Holmes, a benchmark inspired by the reasoning process of Sherlock Holmes, designed to evaluate the complex video reasoning capabilities of MLLMs. Video-Holmes consists of 1,837 questions derived from 270 manually annotated suspense short films, which spans seven carefully designed tasks. Each task is constructed by first identifying key events and causal relationships within films, and then designing questions that require models to actively locate and connect multiple relevant visual clues scattered across different video segments. Our comprehensive evaluation of state-of-the-art MLLMs reveals that, while these models generally excel at visual perception, they encounter substantial difficulties with integrating information and often miss critical clues. For example, the best-performing model, Gemini-2.5-Pro, achieves an accuracy of only 45%, with most models scoring below 40%. We aim that Video-Holmes can serve as a "Holmes-test" for multimodal reasoning, motivating models to reason more like humans and emphasizing the ongoing challenges in this field. The benchmark is released in https://github.com/TencentARC/Video-Holmes.

Figures (12)

• Figure 1: An example of Video-Holmes. Models are required to actively locate and connect multiple relevant visual clues scattered across different video segments to render the final answer.
• Figure 2: Construction and evaluation pipeline of Video-Holmes. We select 270 high-quality suspense short films for human annotation. Next, we design 7 challenging tasks and employ DeepSeek to generate questions. Finally, we evaluate SOTA MLLMs and use DeepSeek to analyze their responses.
• Figure 3: Comparison of question types between Video-Holmes and existing benchmarks. Existing benchmarks primarily involve clue-given questions, where models depend on explicitly provided clues to derive answers. In contrast, Video-Holmes adopts an active seeking paradigm, requiring models to actively locate and connect multiple relevant visual clues scattered across different video segments. (Key frames are marked with black boxes and magnified.)
• Figure 4: Example of model reasoning processes on Video-Holmes. VideoChat-R1 misinterprets visual information, incorrectly perceiving a tattooed man, Qwen2.5-VL overlooks critical visual clues (the baby), Video-R1 fails to establish logical connections between the visual clues, and Intern-VL3-8B guesses the right option with a wrong reasoning process.
• Figure 5: Example of human annotation.