An Empirical Comparison of Video Frame Sampling Methods for Multi-Modal RAG Retrieval

TL;DR

The paper investigates which video-frame sampling methods optimize Video RAG retrieval when answering natural language questions. It conducts a side-by-side evaluation of interval-based, pixel-intensity, structural, semantic feature-based, and shot-boundary methods on MSR-VTT using a 1 FPS master frame set, measuring retrieval via recall at top-k. Key findings show that many sampling schemes achieve recall comparable to storing every frame while using substantially fewer frames and reduced storage and computation, with dynamic thresholds often performing as well as or better than static thresholds. The results emphasize that there is no one-size-fits-all sampling method; performance varies by video category, motivating use-case–driven method selection and potential future enhancements with joint text-frame encodings.

Abstract

Numerous video frame sampling methodologies detailed in the literature present a significant challenge in determining the optimal video frame method for Video RAG pattern without a comparative side-by-side analysis. In this work, we investigate the trade-offs in frame sampling methods for Video & Frame Retrieval using natural language questions. We explore the balance between the quantity of sampled frames and the retrieval recall score, aiming to identify efficient video frame sampling strategies that maintain high retrieval efficacy with reduced storage and processing demands. Our study focuses on the storage and retrieval of image data (video frames) within a vector database required by Video RAG pattern, comparing the effectiveness of various frame sampling techniques. Our investigation indicates that the recall@k metric for both text-to-video and text-to-frame retrieval tasks using various methods covered as part of this work is comparable to or exceeds that of storing each frame from the video. Our findings are intended to inform the selection of frame sampling methods for practical Video RAG implementations, serving as a springboard for innovative research in this domain.

Figures (30)

• Figure 1: Video RAG Overview. Key frames are extracted from the video and encoded as image vectors before saving them into the vector store. The raw/resized frames and original videos are saved in unstructured data storage. The user text query is encoded into a joint vector space (with images) and matched with vector store records to retrieve top-$k$ records. The raw frames (Base64 representation) for top-$k$ best matches are passed back to the multi-modal LLM along with text query and system message to generate the response.
• Figure 2: Sample videos from MSR-VTT
• Figure 3: The distribution of video categories (taken from MSR-VTT)
• Figure 4: Video-level annotation example for MSR-VTT
• Figure 5: Newly created video frame-level annotation example for MSR-VTT