VNU-Bench: A Benchmarking Dataset for Multi-Source Multimodal News Video Understanding

TL;DR

VNU-Bench presents the first benchmark for multi-source, cross-video understanding in news, addressing real-world cross-outlet reasoning by combining a diverse dataset with a novel 10-type question taxonomy. It employs a hybrid human-model QA generation and rigorous quality control to produce 2,501 high-quality cross-source QAs drawn from 429 news groups and 1,405 videos. Evaluations across closed- and open-source MLLMs reveal substantial challenges in cross-source integration, with performance gaps widening on integration tasks and across domains. The work lays a foundation for future research in cross-source multimodal news understanding by providing a scalable dataset, structured taxonomy, and empirical insights into current model capabilities.

Abstract

News videos are carefully edited multimodal narratives that combine narration, visuals, and external quotations into coherent storylines. In recent years, there have been significant advances in evaluating multimodal large language models (MLLMs) for news video understanding. However, existing benchmarks largely focus on single-source, intra-video reasoning, where each report is processed in isolation. In contrast, real-world news consumption is inherently multi-sourced: the same event is reported by different outlets with complementary details, distinct narrative choices, and sometimes conflicting claims that unfold over time. Robust news understanding, therefore, requires models to compare perspectives from different sources, align multimodal evidence across sources, and synthesize multi-source information. To fill this gap, we introduce VNU-Bench, the first benchmark for multi-source, cross-video understanding in the news domain. We design a set of new question types that are unique in testing models' ability of understanding multi-source multimodal news from a variety of different angles. We design a novel hybrid human-model QA generation process that addresses the issues of scalability and quality control in building a large dataset for cross-source news understanding. The dataset comprises 429 news groups, 1,405 videos, and 2,501 high-quality questions. Comprehensive evaluation of both closed- and open-source multimodal models shows that VNU-Bench poses substantial challenges for current MLLMs.

Figures (6)

• Figure 1: Overview of the VNUBench dataset construction pipeline. News video groups are collected from diverse domains, with a small subset manually annotated to serve as few-shot exemplars. Visual frames and transcripts are extracted from the videos and integrated with type-specific prompting, common rules prompting and few-shot examples to synthesize initial QA drafts via MLLMs. Subsequently, the drafts undergo a strict quality control process assisted by an MLLM-as-judge mechanism, which evaluates candidates based on single-source solvability, ambiguity analysis, and difficulty analysis to eliminate trivial samples. Finally, the filtered questions are carefully checked by voluteers to ensure the high quality of the final dataset.
• Figure 2: Illustrative examples of generated question types. The left side shows representative questions of Category 1 (Multi-source News Comparison), which focus on comparing evidence and visuals across videos. The right side shows Category 2 (Cross-source News Integration), requiring cross-video and cross-modal reasoning to detect conflicts and trace temporal developments. Correct answers are highlighted in red. Full examples for all types are provided in Appendix \ref{['apd:sample_qa']}.
• Figure 3: (a) Video length distribution and (b) significant organization distribution of VNU-Bench, where the names have to be zoomed in to see.
• Figure 4: Domain distribution of VNU-Bench.
• Figure 5: Domain analysis: model accuracy on news of different domains.