Parameter-free Video Segmentation for Vision and Language Understanding

TL;DR

This work tackles the challenge of applying vision-language models to long-form videos by introducing MDLSeg, a parameter-free segmentation method that partitions a video into contiguous scenes by minimizing the total description length across scenes. The method uses a dynamic-programming search to compute an optimal partition, with a cost that combines per-scene encoding bits and the number of scenes, enabling robust segmentation without thresholds or predefined scene counts. MDLSeg is integrated into modular systems for long-form movie summarisation and retrieval-augmented video QA, achieving superior downstream performance compared to baselines and several state-of-the-art approaches. The results demonstrate that accurate scene boundaries improve both summarisation quality and QA accuracy, highlighting the practical value of data-driven, parameter-free segmentation for scalable multimodal video understanding.

Abstract

The proliferation of creative video content has driven demand for adapting language models to handle video input and enable multimodal understanding. However, end-to-end models struggle to process long videos due to their size and complexity. An effective alternative is to divide them into smaller chunks to be processed separately, and this motivates a method for choosing where the chunk boundaries should be. In this paper, we propose an algorithm for segmenting videos into contiguous chunks, based on the minimum description length principle, coupled with a dynamic programming search. The algorithm is entirely parameter-free, given feature vectors, not requiring a set threshold or the number or size of chunks to be specified. We show empirically that the breakpoints it produces more accurately approximate scene boundaries in long videos, compared with existing methods for scene detection, even when such methods have access to the true number of scenes. We then showcase this algorithm in two tasks: long video summarization, and retrieval-augmented video question answering. In both cases, scene breaks produced by our algorithm lead to better downstream performance than existing methods for video segmentation.

Figures (4)

• Figure 1: Movie summarization: scene breaks from MDLSeg enable the production of a pseudo-screenplay (centre) from the input video/audio (top left), by first extracting the raw transcript, then using MDLSeg to segment the video into scenes, and generating visual descriptions from each scene (top right). Then from these outputs, and inserted names using the character bank (bottom left), we can summarise hierarchcially (centre right, bottom right).
• Figure 2: Example of a scene break (horizontal line) detected by MDLSeg as it appears in the pseudo-transcript for the movie The Silence of the Lambs (1991). The text shows the transcribed dialogue, with names inferred by our method. The images display visual captions along with keyframes from which they were derived.
• Figure 3: Video question answering: Pipeline of how we use MDLSeg for retrieval-based video question answering. The input (highlighted in pink in the top left), consists of the full video with the accompanying transcript, and a multiple choice question (MCQ). The video is segmented with MDLSeg, feature vectors are computed for each scene and the one with the highest dot product with the MCQ is retrieved. Then, for the retrieved scene, a video model is used to produce a visual description. This description, along with the scene transcript and the question are input to a language-only model to produce an answer.
• Figure 4: Computing the cost of assigning the character Clarice Starling (Jodie Foster) to three different scenes of The Silence of the Lambs (1991). After computing the cost of assigning a character to a each scene, we then compute the cost of assigning a character to a speaker ID as the mean of the cost of assigning them to all scenes that speaker ID appears in.