STEC: A Reference-Free Spatio-Temporal Entropy Coverage Metric for Evaluating Sampled Video Frames

TL;DR

This work tackles the lack of a task-agnostic, reference-free metric for evaluating frame sampling quality in video processing pipelines. It introduces Spatio-Temporal Entropy Coverage (STEC), which combines per-frame spatial information via Spatio-Temporal Frame Entropy (STFE) with temporal coverage and non-redundancy, formalized as $\mathrm{STEC}(\mathcal{S}) = \left( \frac{1}{K} \sum_{j=1}^K E_s(f_{i_j}) \right) \cdot T(\mathcal{S}) \cdot R(\mathcal{S})$ with $T(\mathcal{S}) = E_t(\mathcal{S}) \cdot C_t(\mathcal{S})$. A MSR-VTT experiment set shows STEC clearly differentiates common sampling strategies and reveals per-video robustness patterns not visible in average scores, establishing STEC as a practical diagnostic signal for frame sampling under budget constraints. While not a predictor of downstream task accuracy, higher STEC often correlates with better retrieval performance under fixed encoders, supporting its utility for screening and analyzing sampling strategies in efficient video understanding pipelines. The method is lightweight, training-free, and broadly applicable, with code available to facilitate reproducibility and adoption.

Abstract

Frame sampling is a fundamental component in video understanding and video--language model pipelines, yet evaluating the quality of sampled frames remains challenging. Existing evaluation metrics primarily focus on perceptual quality or reconstruction fidelity, and are not designed to assess whether a set of sampled frames adequately captures informative and representative video content. We propose Spatio-Temporal Entropy Coverage (STEC), a simple and non-reference metric for evaluating the effectiveness of video frame sampling. STEC builds upon Spatio-Temporal Frame Entropy (STFE), which measures per-frame spatial information via entropy-based structural complexity, and evaluates sampled frames based on their temporal coverage and redundancy. By jointly modeling spatial information strength, temporal dispersion, and non-redundancy, STEC provides a principled and lightweight measure of sampling quality. Experiments on the MSR-VTT test-1k benchmark demonstrate that STEC clearly differentiates common sampling strategies, including random, uniform, and content-aware methods. We further show that STEC reveals robustness patterns across individual videos that are not captured by average performance alone, highlighting its practical value as a general-purpose evaluation tool for efficient video understanding. We emphasize that STEC is not designed to predict downstream task accuracy, but to provide a task-agnostic diagnostic signal for analyzing frame sampling behavior under constrained budgets.

Figures (3)

• Figure 1: Temporal entropy versus temporal span. High temporal entropy alone does not guarantee global coverage: frames may be evenly distributed within a narrow temporal segment (top). Combining temporal entropy with a span-based coverage term ensures that sampled frames are both evenly dispersed and cover the full extent of the video timeline (bottom).
• Figure 2: Distribution of per-video STEC scores on MSR-VTT test-1k. Boxplots summarize STEC values across individual videos for different sampling strategies. The median and interquartile range represent typical performance, while outliers correspond to rare extreme cases. Content-aware methods (Katna and STACFP) shift the overall distribution toward higher STEC values, indicating improved robustness across diverse videos, rather than performance dominated by a small number of outlier cases. Notably, STACFP exhibits a more compact distribution with fewer extreme failures, indicating stronger per-video robustness compared to methods with higher variance.
• Figure 3: Qualitative analysis of frame sampling behavior using STEC. Two representative MSR-VTT videos illustrate the coverage--redundancy trade-off captured by STEC. Top: For a video with diverse scenes, Uniform and Random achieve full temporal coverage but select many redundant frames, resulting in low STEC. Content-aware methods improve diversity: STACFP focuses on informative segments with higher non-redundancy, while Katna maintains full coverage and achieves the highest STEC. Bottom: For a video dominated by repetitive content, Uniform and Random repeatedly select near-duplicate frames, leading to very low non-redundancy. Content-aware methods explicitly reduce redundancy; STACFP emphasizes diversity with reduced coverage, whereas Katna balances coverage and diversity and yields the highest STEC.