CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models

TL;DR

CrossVid tackles the problem of cross-video reasoning (CVR) by introducing a first comprehensive benchmark explicitly designed to test spatial-temporal CVR in multimodal large language models.The benchmark spans four reasoning dimensions and ten tasks, comprises 5,331 videos and 9,015 QA pairs, and uses a semi-automated annotation pipeline with expert quality control.Extensive evaluation across 22 MLLMs shows that CVR remains challenging, with the best model achieving only 50.4% accuracy and humans outperforming by a large margin, highlighting gaps in current CVR capabilities.Key findings indicate that closed-source and 'thinking-enabled' models fare better, frame context and chain-of-thought prompting can improve performance in certain tasks, and CrossVid provides guidance for future improvements in multi-video understanding.

Abstract

Cross-Video Reasoning (CVR) presents a significant challenge in video understanding, which requires simultaneous understanding of multiple videos to aggregate and compare information across groups of videos. Most existing video understanding benchmarks focus on single-video analysis, failing to assess the ability of multimodal large language models (MLLMs) to simultaneously reason over various videos. Recent benchmarks evaluate MLLMs' capabilities on multi-view videos that capture different perspectives of the same scene. However, their limited tasks hinder a thorough assessment of MLLMs in diverse real-world CVR scenarios. To this end, we introduce CrossVid, the first benchmark designed to comprehensively evaluate MLLMs' spatial-temporal reasoning ability in cross-video contexts. Firstly, CrossVid encompasses a wide spectrum of hierarchical tasks, comprising four high-level dimensions and ten specific tasks, thereby closely reflecting the complex and varied nature of real-world video understanding. Secondly, CrossVid provides 5,331 videos, along with 9,015 challenging question-answering pairs, spanning single-choice, multiple-choice, and open-ended question formats. Through extensive experiments on various open-source and closed-source MLLMs, we observe that Gemini-2.5-Pro performs best on CrossVid, achieving an average accuracy of 50.4%. Notably, our in-depth case study demonstrates that most current MLLMs struggle with CVR tasks, primarily due to their inability to integrate or compare evidence distributed across multiple videos for reasoning. These insights highlight the potential of CrossVid to guide future advancements in enhancing MLLMs' CVR capabilities.

Figures (28)

• Figure 1: Performance of MLLMs on CrossVid.
• Figure 2: Overview of CrossVid. It evaluates MLLMs' CVR capability on 4 dimensions: comparative analysis, temporal understanding, multi-view reasoning, and free-form QA. It contains 10 distinct tasks: behavioral understanding (BU), narrative comprehension (NC), culinary comparison (CC), procedural error analysis (PEA), plot inference (PI), functional step alignment (FSA), procedural step sequencing (PSS), multi-view spatial reasoning (MSR), multi-view object counting (MOC) and comparative culinary QA (CCQA).
• Figure 3: Statistical analysis of our CrossVid dataset. It consists of 4 high-level dimensions and 10 specific tasks, covering a wide range of video durations and video sources of 7 primary categories and 32 genres.
• Figure 4: Illustration of the CrossVid annotation pipeline. The process consists of the following main stages: (1) Frames are extracted from videos and captioned by Qwen2.5-VL-72B; (2) Deepseek-R1 generates QA pairs using task-specific prompts; (3) The QA pairs undergo rigorous human quality review, including data filtering, refinement, and quality control.
• Figure 5: Captioning prompt for YouCook2 dataset.