VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?

TL;DR

VidComposition tackles the challenge of evaluating MLLMs on fine-grained video composition understanding in compiled videos. It introduces a high-quality, human-annotated benchmark with 982 videos and 1706 QA pairs spanning 15 tasks across five composition aspects, and systematically evaluates 33 MLLMs. The results reveal a large gap between human and model performance, with improvements observed when using higher visual resolution, larger language models, and more training data, though increasing the number of frames yields no consistent gains. The benchmark provides actionable guidance for model design and has potential applications in automated evaluation of generated video quality, offering a benchmarked lens into the next generation of video understanding systems.

Abstract

The advancement of Multimodal Large Language Models (MLLMs) has enabled significant progress in multimodal understanding, expanding their capacity to analyze video content. However, existing evaluation benchmarks for MLLMs primarily focus on abstract video comprehension, lacking a detailed assessment of their ability to understand video compositions, the nuanced interpretation of how visual elements combine and interact within highly compiled video contexts. We introduce VidComposition, a new benchmark specifically designed to evaluate the video composition understanding capabilities of MLLMs using carefully curated compiled videos and cinematic-level annotations. VidComposition includes 982 videos with 1706 multiple-choice questions, covering various compositional aspects such as camera movement, angle, shot size, narrative structure, character actions and emotions, etc. Our comprehensive evaluation of 33 open-source and proprietary MLLMs reveals a significant performance gap between human and model capabilities. This highlights the limitations of current MLLMs in understanding complex, compiled video compositions and offers insights into areas for further improvement. The leaderboard and evaluation code are available at https://yunlong10.github.io/VidComposition/

Figures (14)

• Figure 1: Top MLLMs' performance on VidComposition, across 15 tasks of 5 aspects of video composition understanding: Cinematography Analysis, Character Understanding, Narrative Understanding, Scene Perception, and Making Analysis.
• Figure 2: VidComposition comprises 15 categories of high-quality QA pairs, focusing on five aspects of compositions in compiled videos: cinematography, character, narrative, scene, and making. The correct answers are highlighted.
• Figure 3: (Left) Task statistics in VidComposition, organized into five main categories: Cinematography Analysis (CA), Character Understanding (CU), Narrative Understanding (NU), Scene Perception (SP), and Making Analysis (MA), comprising a total of 15 sub-tasks. The number of QA pairs is shown in parentheses below each task. (Right) The difficulty distribution across these five categories. If a question is answered correctly by more than 60% of MLLMs, it will be labeled as "Easy." Conversely, if a question is answered correctly by fewer than 10% of MLLMs, it will be labeled as "Super Hard."
• Figure 4: #frm analysis. We compare the overall accuracy of models from the same series with the same LLM size and Res.. The results indicate a counterintuitive irrelevance between the overall accuracy and input #frm.
• Figure 5: Qualitative analysis. Green represents correct answers, while red indicates wrong prediction or explanation. More cases can be found in Supplementary.