FrameOracle: Learning What to See and How Much to See in Videos

TL;DR

FrameOracle addresses the inefficiency of processing all video frames by jointly predicting which frames matter and how many are needed for a given query. It features a lightweight, plug-and-play FrameOracle module with a cross-modal fusion encoder and dual heads, trained via a four-stage curriculum that culminates in supervision from FrameOracle-41K, a large VideoQA dataset with ground-truth keyframes. Empirically, FrameOracle reduces input frames from 16 to 10.4 without accuracy loss and from 64 to 13.9 with a small accuracy gain, while also cutting compute and latency, demonstrating state-of-the-art efficiency-accuracy trade-offs for scalable video understanding. The work emphasizes model-agnostic applicability and contributes FrameOracle-41K to enable targeted, minimal evidentiary frame selection across diverse VLMs and benchmarks.

Abstract

Vision-language models (VLMs) have advanced video understanding, but their performance is limited by the number of input frames they can process. Existing frame sampling strategies, such as uniform or fixed-budget selection, often fail to adapt to variations in information density or task complexity, resulting in inefficiency and information loss. To address this, we present FrameOracle, a lightweight and plug-and-play module that predicts both (1) which frames are most relevant to a given query and (2) how many frames are needed. FrameOracle is trained using a four-stage curriculum, with the first three stages relying on weak proxy signals such as cross-modal similarity. In the final stage, it leverages stronger supervision from a new dataset we introduce, FrameOracle-41K, the first large-scale VideoQA collection to provide keyframe annotations specifying the minimal set of frames required to answer each question. Extensive experiments across five VLMs and six benchmarks demonstrate that FrameOracle reduces 16-frame inputs to an average of 10.4 frames without any loss in accuracy. When starting from 64-frame candidates, it reduces the input to an average of 13.9 frames while improving accuracy by 1.4%, achieving state-of-the-art efficiency-accuracy trade-offs for scalable video understanding.

Figures (6)

• Figure 1: FrameOracle-41K data generation pipeline. Stage I (agent-based keyframe mining) iteratively explores each video using a multimodal agent, ultimately returning a predicted answer with confidence and relevance scores for all visited frames. Stage II (filtering and verification) first discards frames with low relevance scores and then verifies sufficiency by requiring three independent VLMs to answer correctly using only the remaining keyframes.
• Figure 2: FrameOracle-41K dataset statistics. (a) Video durations range from 2 to 3 minutes, offering enough temporal context for analysis without including unnecessary frames. (b) Keyframe annotations are sparse: most videos require fewer than 10 frames, yet the dataset also includes harder edge cases to capture more challenging scenarios.
• Figure 3: Overview of the FrameOracle pipeline. FrameOracle (dashed box) receives raw video frames and the textual prompt, and jointly predicts frame importance and the number of frames to keep. It outputs a compact keyframe subset, which is fed into the downstream VLM. $V_C$ denotes the pre-sampled frame collection, and $V_S$ denotes the subset selected by FrameOracle.
• Figure 4: Four-stage training strategy of FrameOracle. The model is progressively optimized from weak to strong supervision, culminating in supervised fine-tuning with FrameOracle-41K annotations. Fire icons indicate trainable modules, while snowflake icons denote frozen ones.
• Figure 5: Qualitative examples. FrameOracle answers correctly while using only a few frames (2 to 4 out of 16), compared to uniform sampling, which relies on the full input.