ViSMaP: Unsupervised Hour-long Video Summarisation by Meta-Prompting

TL;DR

ViSMaP addresses unsupervised hour-long video summarisation by transferring knowledge from annotated short-form videos through a meta-prompting loop over large language models. It first trains a short-form video summariser on 3-minute segments, then iteratively generates and refines hour-long video pseudo-summaries via a generator/evaluator/optimiser LLM trio, and finally adapts the short-form model to long-form data using a noise-robust Symmetric Cross Entropy loss. The method achieves performance comparable to fully supervised state-of-the-art models and demonstrates strong cross-domain generalisation without expensive long-form annotations. This approach offers a scalable framework for long-form video understanding and can be extended with multimodal inputs and hierarchical summarisation for broader applicability.

Abstract

We introduce ViSMap: Unsupervised Video Summarisation by Meta Prompting, a system to summarise hour long videos with no-supervision. Most existing video understanding models work well on short videos of pre-segmented events, yet they struggle to summarise longer videos where relevant events are sparsely distributed and not pre-segmented. Moreover, long-form video understanding often relies on supervised hierarchical training that needs extensive annotations which are costly, slow and prone to inconsistency. With ViSMaP we bridge the gap between short videos (where annotated data is plentiful) and long ones (where it's not). We rely on LLMs to create optimised pseudo-summaries of long videos using segment descriptions from short ones. These pseudo-summaries are used as training data for a model that generates long-form video summaries, bypassing the need for expensive annotations of long videos. Specifically, we adopt a meta-prompting strategy to iteratively generate and refine creating pseudo-summaries of long videos. The strategy leverages short clip descriptions obtained from a supervised short video model to guide the summary. Each iteration uses three LLMs working in sequence: one to generate the pseudo-summary from clip descriptions, another to evaluate it, and a third to optimise the prompt of the generator. This iteration is necessary because the quality of the pseudo-summaries is highly dependent on the generator prompt, and varies widely among videos. We evaluate our summaries extensively on multiple datasets; our results show that ViSMaP achieves performance comparable to fully supervised state-of-the-art models while generalising across domains without sacrificing performance. Code will be released upon publication.

Figures (5)

• Figure 1: Two main challenges we address with our approach: (a) Bridging the semantic gap between short-form segment descriptions and hour-long video summaries descriptions (b) Overcoming the domain shift between the source domain and the target domain.
• Figure 2: Motivation of our VisMaP. Most existing video summarisation models focus on minute-level short-form videos, while hour-long videos, which are more common in real-world scenarios, are often overlooked due to their length, content complexity, and the prohibitively high cost of manual annotation. We propose a cross-domain unsupervised approach for hour-long video summarisation. It leverages the inductive power of multiple LLMs to generate high-quality pseudo-summaries from short video segments via meta-prompting. These pseudo-summaries are then used to train a model, enabling effective summarisation of long videos without costly human annotations.
• Figure 3: An overview of our VisMaP. (a) First stage: we use 180-second source video $v^s$ for supervised pretraining to establish basic summary capabilities. (b) Second stage: we split hour-long target videos $v^t$ into 3-minute segments set $\mathbf{V}_i^t$ and process them through the first-stage summary model to generate pseudo captions $\widehat{C}^t$. $\widehat{C}^t$ are then refined through a meta-prompting process with $K$ iterations, using Gemini as the evaluator and GPT-3.5 as the optimiser and the generator, to create more tailored prompts ${Pr}^t$ and summaries $\widehat{Y}^t$. (c) Third stage: Refined $\widehat{Y}^t$ pseudo-summaries are utilised to fine-tune the summary model for effective hour-long video summary.
• Figure 4: An example of summaries from ViSMaP on the Ego4D-HCap dataset.
• Figure 5: Qualitative Results on Ego4D dataset.