MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval

TL;DR

MultiVENT 2.0 introduces a massive, multilingual, event-centric video retrieval benchmark with over 218k videos and around 3.9k hand-crafted queries spanning six languages. The dataset expands prior efforts by integrating InternVid content, providing rich cross-modal signals (visual, audio, embedded text, and video metadata) and a diverse set of event types for realistic retrieval scenarios. The paper details a two-pronged query creation pipeline, robust relevance annotation (including both gold and silver judgments) and comprehensive baselines (VLMs and single-modality pipelines) to highlight current gaps in cross-modal event retrieval. Results show state-of-the-art vision-language models struggle with this task, underscoring the need for more robust, multimodal systems capable of leveraging multiple modalities and multilingual signals for real-world event understanding and retrieval.

Abstract

Efficiently retrieving and synthesizing information from large-scale multimodal collections has become a critical challenge. However, existing video retrieval datasets suffer from scope limitations, primarily focusing on matching descriptive but vague queries with small collections of professionally edited, English-centric videos. To address this gap, we introduce $\textbf{MultiVENT 2.0}$, a large-scale, multilingual event-centric video retrieval benchmark featuring a collection of more than 218,000 news videos and 3,906 queries targeting specific world events. These queries specifically target information found in the visual content, audio, embedded text, and text metadata of the videos, requiring systems leverage all these sources to succeed at the task. Preliminary results show that state-of-the-art vision-language models struggle significantly with this task, and while alternative approaches show promise, they are still insufficient to adequately address this problem. These findings underscore the need for more robust multimodal retrieval systems, as effective video retrieval is a crucial step towards multimodal content understanding and generation.

Figures (6)

• Figure 1: Example query/video pairs from MSR-VTTxu2016msrvtt and MultiVENT 2.0. MSR-VTT primarily contains broad descriptive queries mapped to general English-centric video clips, while MultiVENT 2.0 targets specific current events covering 4 media formats, 6 languages, and subjects like natural disasters, politics, sports, social gatherings, and science.
• Figure 2: Query creation process for event-centric videos within our distractor collection from InternVid. Annotators first create a Base Event query based on the primary event depicted in the video. They then write up to three additional queries focusing on specific and unique aspects of the event: the Description query uses only information from the human-written text description, the Speech query relies on spoken content from the video, and the Embedded Text query utilizes text visible within the video frames.
• Figure 3: Breakdowns of of the number of queries mapped to relevant videos. Figure \ref{['fig:language_train']} shows that MultiVENT Train contains queries targeting the five primary languages from MultiVENT 1.0. On the other hand, in Figure \ref{['fig:language_test']} we see that MultiVENT Test adds queries targeting Spanish events to challenge systems' multilingual robustness. Figure \ref{['fig:video_type']} shows that MultiVENT 2.0 targets videos ranging from professional news broadcasts to raw first-person footage of events. Finally, as seen in Figure \ref{['fig:event_type']}, events in MultiVENT 2.0 generally map to the same categories as MultiVENT 1.0, with a long tail of infrequent event types.
• Figure 4: Example and Label Studio interface for video classification annotations and base query creation i.e., the phrase that represents the "Wikipedia title" of a current event.
• Figure 5: Example and Label Studio interface for writing queries targeting specific aspects of events. For each question, we ask annotators to only use the text description, audio, and embedded text, respectively.