CVBench: Benchmarking Cross-Video Synergies for Complex Multimodal Reasoning

TL;DR

CVBench introduces the first diagnostic benchmark for cross-video relational reasoning in multimodal models, addressing a critical gap in evaluating inter-video object, event, and complex reasoning across multiple videos. It provides 1,000 expert-annotated QA pairs across five domains, a robust QA generation and quality-control pipeline, and a multi-domain dataset designed to stress cross-video integration. Extensive evaluation of 10+ MLLMs reveals large gaps to human performance, with notable weaknesses in inter-video memory and entity disambiguation, and a pronounced divide between closed- and open-source systems. The benchmark offers actionable architectural guidance, including cross-video memory mechanisms and grounded spatiotemporal reasoning, and is publicly available to accelerate progress in robust multi-video understanding.

Abstract

While multimodal large language models (MLLMs) exhibit strong performance on single-video tasks (e.g., video question answering), their capability for spatiotemporal pattern reasoning across multiple videos remains a critical gap in pattern recognition research. However, this capability is essential for real-world applications, including multi-camera surveillance and cross-video procedural learning. To bridge this gap, we present CVBench, the first diagnostic benchmark designed to assess cross-video relational reasoning rigorously. CVBench comprises 1,000 question-answer pairs spanning three hierarchical tiers: cross-video object association (identifying shared entities), cross-video event association (linking temporal or causal event chains), and cross-video complex reasoning (integrating commonsense and domain knowledge). Built from five domain-diverse video clusters (e.g., sports, life records), the benchmark challenges models to analyze and integrate spatiotemporal patterns from dynamic visual streams. Extensive evaluation of 10+ leading MLLMs (including GPT-4o, Gemini-2.0-flash, Qwen2.5-VL) under zero-shot or chain-of-thought prompting paradigms. Key findings reveal stark performance gaps: even top models, such as GPT-4o, achieve only 63.5% accuracy on causal reasoning tasks, compared to the 91.3% accuracy of human performance. Crucially, our analysis reveals fundamental bottlenecks inherent in current MLLMs architectures, notably deficient inter-video context retention and poor disambiguation of overlapping entities. CVBench establishes a rigorous framework for advancing pattern recognition methodologies in multi-video scenarios, providing architectural insights for next-generation models. The data and evaluation code are available at: https://github.com/Hokhim2/CVBench.

Figures (6)

• Figure 1: Illustration of CVBench task definition. CVBench focuses on three core tasks that are crucial for real-world applications: a) cross-video object association (identifying shared entities across different videos), b) cross-video event association (linking temporal or causal event chains that span across video sequences), and c) cross-video complex reasoning (integrating commonsense to understand and reason about complex interrelationships between video content). The benchmark format is multi-choice QA, and here we only show the correct answer for visualization.
• Figure 2: CVBench statistics and task categories. CVBench comprises three primary task categories and 15 video sub-categories across diverse video domains. We present the distribution of video durations, the number of videos, and representative domains.
• Figure 3: The pipeline for question-answer (QA) pairs annotations in CVBench. The QA generation pipeline is streamlined as four steps: video segment captioning, relational summary annotation, question-answer generation and quality control.
• Figure 4: Impact of video number and video duration.
• Figure 5: Qualitative cases of multi-video reasoning on CVBench based on Video-R1.