MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence

TL;DR

This paper introduces MUVR, a benchmark and task for Multi-modal Untrimmed Video Retrieval on long-video platforms. It proposes a practical retrieval paradigm that combines video-centric multi-modal queries with text as auxiliary input, and organizes data by multi-level visual correspondence across five partitions. The benchmark provides three evaluation regimes (Base, Filter, QA) and a novel Reranking Score to assess MLLMs, with extensive experiments showing current models struggle with untrimmed videos and multimodal queries. The work highlights key gaps in video retrieval and MLLMs, offering a clear direction for advancing fusion, temporal modeling, and efficient reranking in real-world settings.

Abstract

We propose the Multi-modal Untrimmed Video Retrieval task, along with a new benchmark (MUVR) to advance video retrieval for long-video platforms. MUVR aims to retrieve untrimmed videos containing relevant segments using multi-modal queries. It has the following features: 1) Practical retrieval paradigm: MUVR supports video-centric multi-modal queries, expressing fine-grained retrieval needs through long text descriptions, video tag prompts, and mask prompts. It adopts a one-to-many retrieval paradigm and focuses on untrimmed videos, tailored for long-video platform applications. 2) Multi-level visual correspondence: To cover common video categories (e.g., news, travel, dance) and precisely define retrieval matching criteria, we construct multi-level visual correspondence based on core video content (e.g., news events, travel locations, dance moves) which users are interested in and want to retrieve. It covers six levels: copy, event, scene, instance, action, and others. 3) Comprehensive evaluation criteria: We develop 3 versions of MUVR (i.e., Base, Filter, QA). MUVR-Base/Filter evaluates retrieval models, while MUVR-QA assesses MLLMs in a question-answering format. We also propose a Reranking Score to evaluate the reranking ability of MLLMs. MUVR consists of 53K untrimmed videos from the video platform Bilibili, with 1,050 multi-modal queries and 84K matches. Extensive evaluations of 3 state-of-the-art video retrieval models, 6 image-based VLMs, and 10 MLLMs are conducted. MUVR reveals the limitations of retrieval methods in processing untrimmed videos and multi-modal queries, as well as MLLMs in multi-video understanding and reranking. Our code and benchmark is available at https://github.com/debby-0527/MUVR.

Figures (9)

• Figure 1: Visualization of videos from different partitions of MUVR. The two videos are matched due to multi-level visual correspondence. Please refer to the Appendix for more results.
• Figure 2: Illustration of video category breakdown for five partitions that make up MUVR.
• Figure 3: Illustration of MUVR-QA, Reranking Score, and MLLMs prompting.
• Figure 4: Visualization of three relevant videos on the News partition.
• Figure 5: Visualization of three relevant videos on the Region partition. The third video comes from a computer game and brings more challenges.