BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding

TL;DR

BOLT tackles the bottleneck of long-form video understanding in large vision-language models by introducing a training-free, query-guided frame-selection mechanism. The core method uses inverse transform sampling to preferentially select frames that are relevant to the given query while preserving temporal diversity, enabling better use of the model's context window without retraining. The authors validate their approach with ground-truth segment and multi-source retrieval evaluators, showing consistent performance gains across multiple benchmarks (e.g., Video-MME and MLVU) and off-the-shelf VLMs. The findings highlight the practical significance of frame-level prioritization for scalable long-form video reasoning, and they provide a simple, cost-effective way to enhance existing VLMs in real-world settings.

Abstract

Large video-language models (VLMs) have demonstrated promising progress in various video understanding tasks. However, their effectiveness in long-form video analysis is constrained by limited context windows. Traditional approaches, such as uniform frame sampling, often inevitably allocate resources to irrelevant content, diminishing their effectiveness in real-world scenarios. In this paper, we introduce BOLT, a method to BOost Large VLMs without additional Training through a comprehensive study of frame selection strategies. First, to enable a more realistic evaluation of VLMs in long-form video understanding, we propose a multi-source retrieval evaluation setting. Our findings reveal that uniform sampling performs poorly in noisy contexts, underscoring the importance of selecting the right frames. Second, we explore several frame selection strategies based on query-frame similarity and analyze their effectiveness at inference time. Our results show that inverse transform sampling yields the most significant performance improvement, increasing accuracy on the Video-MME benchmark from 53.8% to 56.1% and MLVU benchmark from 58.9% to 63.4%. Our code is available at https://github.com/sming256/BOLT.

Figures (5)

• Figure 1: Given a long video, conventional VLMs adopt uniform sampling during inference, which may divert VLM’s focus due to the distracting backgrounds. In contrast, our query-guided frame sampling can identify and prioritize question-relevant frames, boosting the VLM's performance without requiring fine-tuning.
• Figure 2: (a) Proposed training-free frame selection framework. Without exhaustive fine-tuning of VLMs, their performance can be improved by selecting query-related frames. (b) Top-K selection often yields redundant frames with limited diversity. (c) Watershed grouping may overlook broader temporal context. (d) Inverse transform sampling can prioritize relevant frames while preserving selection diversity, achieving stronger performance on benchmarks, particularly for videos with noisy backgrounds.
• Figure 3: Visualization of frame selection results using inverse transform sampling. The blue curve represents the similarity scores across the entire video sequence, and the red lines indicate the selected frames.
• Figure 4: When visual-query similarity scores remain similar across the video, inverse transform sampling simplifies to uniform sampling.
• Figure 5: Additional visualization results.