MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs

TL;DR

MVU-Eval introduces the first benchmark dedicated to multi-video understanding for Multimodal LLMs, addressing the limitations of single-video benchmarks. It provides 1,824 QA pairs across 4,959 videos to evaluate eight core perception and reasoning competencies, using automated generation and rigorous human QC. The study benchmarks a wide range of open- and closed-source models, revealing substantial gaps, scaling trends, and sensitivity to input modalities and formats. It also outlines concrete future directions for cross-video alignment, spatial reasoning, and scalable fusion to advance robust multi-video understanding in real-world scenarios such as autonomous driving and sports analytics.

Abstract

The advent of Multimodal Large Language Models (MLLMs) has expanded AI capabilities to visual modalities, yet existing evaluation benchmarks remain limited to single-video understanding, overlooking the critical need for multi-video understanding in real-world scenarios (e.g., sports analytics and autonomous driving). To address this significant gap, we introduce MVU-Eval, the first comprehensive benchmark for evaluating Multi-Video Understanding for MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies through 1,824 meticulously curated question-answer pairs spanning 4,959 videos from diverse domains, addressing both fundamental perception tasks and high-order reasoning tasks. These capabilities are rigorously aligned with real-world applications such as multi-sensor synthesis in autonomous systems and cross-angle sports analytics. Through extensive evaluation of state-of-the-art open-source and closed-source models, we reveal significant performance discrepancies and limitations in current MLLMs' ability to perform understanding across multiple videos. The benchmark will be made publicly available to foster future research.

Figures (8)

• Figure 1: Illustration of several representative examples in our MVU-Eval.
• Figure 2: The overall data construction pipeline of MVU-Eval.
• Figure 3: The histogram of #videos.
• Figure 4: The distribution of video categories in MVU-Eval.
• Figure 5: Model scaling of MLLMs on MVU-Eval.