Event-Anchored Frame Selection for Effective Long-Video Understanding

TL;DR

Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline that partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events and ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity.

Abstract

Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.

Figures (7)

• Figure 1: A conceptual illustration of our Event-anchored Frame Selection (EFS) in contrast to the flat sampling baseline. Flat sampling often yields redundant frames and misses key events, causing LVLMs to fail, while EFS partitions videos into events, selects query-relevant anchors, and performs anchor-guided global refinement to enable correct chronological reasoning.
• Figure 2: An illustration of our Event-Anchored Frame Selection (EFS) pipeline. Given a user query, such as "How many times does the instructor appear in different scenarios?", EFS operates in distinct stages: (1) It first acquires visual signals (from DinoV2) and semantic relevance scores (from BLIP2-ITM). (2) The visual signals are used to partition the video into coherent events at local minima of temporal similarity. (3) Within each event, it localizes the most query-relevant frame as an anchor. (4) This initial set of anchors is then globally refined via Adaptive MMR to form the final keyframe set. (5) Finally, this curated set enables the Large Vision-Language Models to accurately answer the query, correctly identifying the four appearances.
• Figure 3: A quantitative comparison on VideoMME using LLaVA-Video-7B, where EFS consistently delivers the largest accuracy gains across 8/16/32/64-frame budgets over different sampling strategies. The text to the left of each bar represents the baseline performance for that method, while the text on the right indicates the corresponding gain.
• Figure 4: Video-based question answering accuracy (%) with different numbers of input frames.
• Figure 5: Qualitative comparison of our EFS versus the uniform sampling. The uniform baseline fails by missing key events, while EFS explicitly identifies the video’s underlying event structure, enabling accurate reasoning and correct answers. Please zoom in for details.