ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search

TL;DR

ShotFinder tackles open-domain video shot retrieval by introducing a benchmark and an imagination-driven three-stage pipeline that expands shot descriptions into video-level queries, retrieves candidate videos from the web, and localizes target shots using description guidance. The benchmark quantifies five single-factor constraints (Temporal, Color, Visual Style, Audio, and Resolution) across 1,210 YouTube-derived samples, enabling controlled analysis of retrieval and grounding under realistic editing scenarios. Experiments show a large gap between current multimodal LLMs and human performance, with temporal localization being relatively easier than color or style and with model scale not guaranteeing universal gains. This work provides practical insights for building open-domain video search systems and guides future directions toward more robust, constraint-aware video understanding.

Abstract

In recent years, large language models (LLMs) have made rapid progress in information retrieval, yet existing research has mainly focused on text or static multimodal settings. Open-domain video shot retrieval, which involves richer temporal structure and more complex semantics, still lacks systematic benchmarks and analysis. To fill this gap, we introduce ShotFinder, a benchmark that formalizes editing requirements as keyframe-oriented shot descriptions and introduces five types of controllable single-factor constraints: Temporal order, Color, Visual style, Audio, and Resolution. We curate 1,210 high-quality samples from YouTube across 20 thematic categories, using large models for generation with human verification. Based on the benchmark, we propose ShotFinder, a text-driven three-stage retrieval and localization pipeline: (1) query expansion via video imagination, (2) candidate video retrieval with a search engine, and (3) description-guided temporal localization. Experiments on multiple closed-source and open-source models reveal a significant gap to human performance, with clear imbalance across constraints: temporal localization is relatively tractable, while color and visual style remain major challenges. These results reveal that open-domain video shot retrieval is still a critical capability that multimodal large models have yet to overcome.

Figures (7)

• Figure 1: Comparison of ShotFinder benchmark with (a) text benchmarks and (b) other multimodal benchmarks. ShotFinder targets open-domain video retrieval. It requires the model to perform "Video Imagination" to bridge the gap between shot descriptions and full video search, followed by searching for URLs and grounding shots.
• Figure 2: ShotFinder construction pipeline. (a) Model-based Description Generation, where Gemini generates shot descriptions with five specific constraints (Resolution, Temporal, Color, Style, Audio); and (b) Human Verification and Refinement, where experts refine the data to ensure semantic accuracy and constraint consistency.
• Figure 3: Illustration of the ShotFinder method pipeline. (1) Generator, utilizing "Video Imagination" to expand shot descriptions into effective search queries; (2) Retriever, fetching and filtering candidate videos from the web; and (3) Localizer, employing MLLMs with adaptive frame sampling to precisely locate the target shot.
• Figure 4: Results of further analysis. Notation ($\mathbf{M \times N}$) : $M =$ Search Queries, $N =$ Candidate URLs per Query. Sampling ($\mathbf{X - Y - Z}$) : Frames sampled for video durations of $<3$ min, $3-10$ min, and $>10$ min respectively.
• Figure 5: Theme distribution, task proportion, and time proportion of the dataset.