Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frame Sampling Strategies Matter: A Benchmark for small vision language models

Marija Brkic, Anas Filali Razzouki, Yannis Tevissen, Khalil Guetari, Mounim A. El Yacoubi

Abstract

Comparing vision language models on videos is particularly complex, as the performances is jointly determined by the model's visual representation capacity and the frame-sampling strategy used to construct the input. Current video benchmarks are suspected to suffer from substantial frame-sampling bias, as models are evaluated with different frame selection strategies. In this work, we propose the first frame-accurate benchmark of state-of-the-art small VLMs for video question-answering, evaluated under controlled frame-sampling strategies. Our results confirm the suspected bias and highlight both data-specific and task-specific behaviors of SVLMs under different frame-sampling techniques. By open-sourcing our benchmarking code, we provide the community with a reproducible and unbiased protocol for evaluating video VLMs and emphasize the need for standardized frame-sampling strategies tailored to each benchmarking dataset in future research.

Frame Sampling Strategies Matter: A Benchmark for small vision language models

Abstract

Comparing vision language models on videos is particularly complex, as the performances is jointly determined by the model's visual representation capacity and the frame-sampling strategy used to construct the input. Current video benchmarks are suspected to suffer from substantial frame-sampling bias, as models are evaluated with different frame selection strategies. In this work, we propose the first frame-accurate benchmark of state-of-the-art small VLMs for video question-answering, evaluated under controlled frame-sampling strategies. Our results confirm the suspected bias and highlight both data-specific and task-specific behaviors of SVLMs under different frame-sampling techniques. By open-sourcing our benchmarking code, we provide the community with a reproducible and unbiased protocol for evaluating video VLMs and emphasize the need for standardized frame-sampling strategies tailored to each benchmarking dataset in future research.

Paper Structure

This paper contains 13 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Frame sampling strategies
  4. Standard sampling
  5. MaxInfo
  6. CSTA
  7. Proposed benchmark
  8. Inference setup
  9. Results
  10. Performance Analysis
  11. Runtime Analysis
  12. Discussion
  13. Conclusion

Figures (4)

  • Figure 1: Comparison between results reported in the original papers and our re-evaluation under a unified sampling setup for FPS uniform sampling.
  • Figure 2: Evolution of Qwen2.5 accuracy on VideoMME when changing the maximum number of frames used after each sampling strategy
  • Figure 3: Distribution of Qwen2.5 performances over all VideoMME tasks with various frame sampling strategies.
  • Figure 4: Impact of maximum frame limit on inference time