From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding

TL;DR

This work tackles the inefficiency of processing long videos by introducing a caption-aware frame scoring framework. It couples STACFP, a spatio-temporal clustering method that yields diverse, temporally coherent frame proposals, with KeyScore, a hybrid scoring mechanism that jointly optimizes semantic alignment, temporal coverage, and contextual impact to select informative frames. Empirical results on MSRVTT, MSVD, and DiDeMo show up to 99% frame reduction while achieving state-of-the-art performance in zero-shot retrieval, keyframe extraction, and action classification, demonstrating both efficiency and accuracy gains. The approach enables scalable, caption-grounded video understanding suitable for large-scale video-language systems and video encoders.

Abstract

Selecting informative keyframes is critical for efficient video understanding, yet existing approaches often rely on heuristics, ignore semantics, or produce redundant frames. We propose KeyScore, a caption-aware frame scoring method that combines three complementary signals: semantic similarity to captions, temporal representativeness, and contextual drop impact. Applied to large-scale video-caption datasets, KeyScore generates frame-level importance scores that enable training keyframe extractors or guiding video-language models. To support this, we also propose STACFP, a Spatio-Temporal Adaptive Clustering method that generates diverse and compact frame proposals across long videos. Together, KeyScore and STACFP reduce uninformative frames while preserving critical content, resulting in faster and more accurate inference. Our experiments on three standard video-language benchmarks (MSRVTT, MSVD, DiDeMo) show that combining STACFP and KeyScore enables up to 99% frame reduction compared to full-frame processing, while outperforming uniform 8-frame encoders in video-text retrieval, keyframe extraction, and action recognition tasks. By focusing on semantically relevant frames, our method enhances both efficiency and performance, enabling scalable and caption-grounded video understanding.

Figures (4)

• Figure 1: Motivating example of our frame scoring. Given the caption "a comedian actor talking in a cloths shop", our method selects keyframes that are semantically aligned with the caption (e.g., actor speaking), while avoiding irrelevant or repetitive frames (e.g., storefront, similar poses).
• Figure 2: End-to-end pipeline of our proposed approach. STACFP first generates candidate keyframes from the input video. Caption and frame embeddings are then extracted using a text encoder and a vision encoder. The frame scoring module (KeyScore) integrates semantic similarity, temporal representation, and contextual drop impact to assign scores to each frame. Finally, task-dependent adaptive thresholding selects the most representative frames for downstream tasks such as retrieval, classification, or summarization.
• Figure 3: Qualitative comparison of different frame proposal methods. UFP (Uniform Frame Proposal) samples frames at regular intervals without considering visual or temporal context, often leading to redundancy and suboptimal coverage (e.g., multiple similar frames during the swing motion). SCFP (Spatial Visual Clustering Frame Proposal) improves diversity via K-means clustering on HSV-based low-level visual features but lacks temporal awareness, resulting in over-sampling static periods. STACFP (Spatio-Temporal Adaptive Clustering Frame Proposal) combines visual and temporal cues for better coverage of key moments with fewer redundant frames. Notably, STACFP captures the start, middle, and end of the golf swing more effectively, highlighting its ability to preserve action dynamics.
• Figure 4: Qualitative examples of KeyScore frame scoring across diverse scenarios. Each example shows (top) the final KeyScore curve with top keyframes highlighted, (middle) uniformly sampled frames with scores, and (bottom) individual scoring components. The semantic similarity score (S) reliably highlights frames that directly align with the caption, ensuring semantic grounding (prosthetic setup in (a), mountain landscapes in (b), actor in (c), and Minnie Mouse in (d)). The contextual drop impact score (D) emphasizes indispensable frames whose removal significantly degrades video–text similarity, ensuring key evidence is preserved. The temporal representativeness score (T) favors frequently recurring frames, which supports temporal coverage but may introduce redundancy or less informative content. Combining all three signals yields compact sets of keyframes that maximize semantic relevance and contextual saliency while maintaining temporal diversity. Examples are taken from MSR-VTT msrvtt.