Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models

TL;DR

Scene-VLM introduces a fine-tuned vision-language model for video scene segmentation that fuses multimodal shot representations (frames, transcripts, and metadata) in a sequential, context-aware framework. A context–focus window enables robust temporal reasoning, while a token-logit-based confidence scheme provides flexible precision–recall control and supports post-hoc explanations through lightweight supervision. The approach achieves state-of-the-art results on MovieNet, demonstrates strong zero-shot generalization to BBC Planet Earth, and adapts to the related task of video chaptering with competitive performance. Together with extensive ablations and explainability analyses, Scene-VLM highlights the value of end-to-end multimodal grounding and narrative reasoning for scalable, interpretable video understanding.

Abstract

Segmenting long-form videos into semantically coherent scenes is a fundamental task in large-scale video understanding. Existing encoder-based methods are limited by visual-centric biases, classify each shot in isolation without leveraging sequential dependencies, and lack both narrative understanding and explainability. In this paper, we present Scene-VLM, the first fine-tuned vision-language model (VLM) framework for video scene segmentation. Scene-VLM jointly processes visual and textual cues including frames, transcriptions, and optional metadata to enable multimodal reasoning across consecutive shots. The model generates predictions sequentially with causal dependencies among shots and introduces a context-focus window mechanism to ensure sufficient temporal context for each shot-level decision. In addition, we propose a scheme to extract confidence scores from the token-level logits of the VLM, enabling controllable precision-recall trade-offs that were previously limited to encoder-based methods. Furthermore, we demonstrate that our model can be aligned to generate coherent natural-language rationales for its boundary decisions through minimal targeted supervision. Our approach achieves state-of-the-art performance on standard scene segmentation benchmarks. On MovieNet, for example, Scene-VLM yields significant improvements of +6 AP and +13.7 F1 over the previous leading method.

Figures (11)

• Figure 1: Video scene segmentation with Scene-VLM. We present Scene-VLM, the first vision-language model (VLM) framework fine-tuned for video scene segmentation. Scene-VLM jointly processes visual frames, dialogue, and metadata from consecutive shots to sequentially predict scene boundaries with associated confidence scores, and can be aligned to produce coherent post-hoc explanations for its decisions.
• Figure 2: Proposed approach. The VLM receives a sequence of $N$ multimodal shot representations as input. Each shot representation consists of visual frames, dialogue, and optional metadata. Scene-VLM processes the shots within a focus window (green) using information from a larger context window (gray), and outputs scene boundary predictions for the shots in focus in the format "Shot $i$: Yes/No." For each shot, we compute a confidence score by probing the softmax logits of the "Yes" and "No" tokens and normalize by $P(\texttt{Yes})/(P(\texttt{Yes}) + P(\texttt{No}))$. The model can also be aligned to generate coherent post-hoc explanations for its scene-boundary decisions.
• Figure 3: Attention by modality. Visualization of attention distribution across input modalities (visual, subtitles, and actor IDs) as well as preceding output shot predictions. (a) Summed attention reveals strong visual dominance and high dependency on prior output tokens. (b) Averaged (length-normalized) attention highlights that subtitles and actor IDs contribute comparably to visual tokens.
• Figure 4: Attention distributions across shots and modalities. Figures show modality-level stacked attention shares for three shot predictions: (a) Shot 7, (b) Shot 11, and (c) Shot 15. Attention is computed between the output token of the corresponding shot prediction and the input tokens for all shots. Each bar represents the relative attention of the visual, subtitle, and actor-ID input modalities per shot.
• Figure 5: Focus mechanism prevents edge degradation. Performance collapses at boundaries without focus mechanism (red) but remains stable with focus mechanism (blue).