LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning

TL;DR

This work tackles the inefficiency of naively encoding all video frames for detailed captioning by introducing a Learnable Frame Selector (LFS) that achieves event-aware and temporally diverse frame sampling. LFS combines a temporal importance model (TSNet), a stratified Top-$K$ selection scheme, and caption-guided optimization using a frozen video-LLM to directly maximize caption quality without updating the language model. It also introduces ICH-CC, a human-cognition–driven benchmark for detailed captions and questions on intangible cultural heritage cuisine videos. Across multiple backbones and benchmarks, including VDC, Dream-1K, and zero-shot video QA, LFS yields consistent gains, demonstrating that effective frame selection substantially enhances downstream captioning and reasoning tasks.

Abstract

Video captioning models convert frames into visual tokens and generate descriptions with large language models (LLMs). Since encoding all frames is prohibitively expensive, uniform sampling is the default choice, but it enforces equal temporal coverage while ignoring the uneven events distribution. This motivates a Learnable Frame Selector (LFS) that selects temporally diverse and event-relevant frames. LFS explicitly models temporal importance to balance temporal diversity and event relevance, and employs a stratified strategy to ensure temporal coverage while avoiding clustering. Crucially, LFS leverages caption feedback from frozen video-LLMs to learn frame selection that directly optimizes downstream caption quality. Additionally, we identify the gap between existing benchmark and human's cognition. Thus, we introduce ICH-CC built from carefully designed questions by annotators that reflect human-consistent understanding of video. Experiments indicate that LFS consistently improves detailed video captioning across two representative community benchmarks and ICH-CC, achieving up to 2.0% gains on VDC and over 4% gains on ICH-CC. Moreover, we observe that enhanced captions with LFS leads to improved performance on video question answering. Overall, LFS provides an effective and easy-to-integrate solution for detailed video captioning.

Figures (5)

• Figure 1: Performance comparison between baselines and the LFS-enhanced counterparts across nine benchmarks. Baselines include AuroraCap-7B, Tarsier2-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B.
• Figure 2: The overall scheme of the proposed learnable frame selector (LFS). (a) and (b) are the training and inference process of LFS, respectively.
• Figure 3: The architecture of temporal scoring network (TSNet).
• Figure 4: Overview of ICH-CC construction pipeline. The five annotators cross-checked each other’s annotated data. After five rounds, each data was thoroughly examined by all annotators to eliminate human biases.
• Figure 5: Qualitative results on the ICH-CC-en benchmark. The left shows Qwen3-VL-8B using uniform sampling and our LFS. ☆ indicates selected event-aware frames, and the bar chart visualizes their distribution along the timeline.