SummDiff: Generative Modeling of Video Summarization with Diffusion

TL;DR

SummDiff reframes video summarization as conditional generative modeling to capture the subjective distribution of plausible summaries. It uses a diffusion-based denoiser conditioned on video features, quantizes scores with a learnable codebook, and applies a knapsack-based, clip-level selection (via Kernel Temporal Segmentation) to produce final summaries. The approach achieves state-of-the-art results on TVSum, SumMe, and Mr. HiSum, and introduces knapsack-inspired metrics (CIS, WIR, WSE) to better evaluate the faithfulness of predicted scores to the final summary. By enabling sampling of multiple summaries per video, SummDiff better reflects diverse human perspectives and provides a scalable, principled framework for subjective video summarization.

Abstract

Video summarization is a task of shortening a video by choosing a subset of frames while preserving its essential moments. Despite the innate subjectivity of the task, previous works have deterministically regressed to an averaged frame score over multiple raters, ignoring the inherent subjectivity of what constitutes a good summary. We propose a novel problem formulation by framing video summarization as a conditional generation task, allowing a model to learn the distribution of good summaries and to generate multiple plausible summaries that better reflect varying human perspectives. Adopting diffusion models for the first time in video summarization, our proposed method, SummDiff, dynamically adapts to visual contexts and generates multiple candidate summaries conditioned on the input video. Extensive experiments demonstrate that SummDiff not only achieves the state-of-the-art performance on various benchmarks but also produces summaries that closely align with individual annotator preferences. Moreover, we provide a deeper insight with novel metrics from an analysis of the knapsack, which is an important last step of generating summaries but has been overlooked in evaluation.

Figures (9)

• Figure 1: Overview of SummDiff. Given an input video, SummDiff generates importance scores conditioned on video frames. $T$ denotes the number of DDIM steps.
• Figure 2: Video Importance Score Denoiser. We use AdaLN layer to inject time and positional conditions, following peebles2023scalable.
• Figure 3: Ratio of summaries with $\tau \ge 0.25$ for each video-annotator pair. The heatmap illustrates how closely each method’s summary matches with individual annotations. SummDiff covers a larger area of the heatmap, which indicates better coverage over various summaries.
• Figure 4: Distribution of true and predicted summaries of given videos. SummDiff can generate various summaries and cover the targeted distribution of summaries, while baselines deterministically predicts a single summary.
• Figure 5: Demonstration of video summaries generated by competing methods on a TVSum video. Shaded parts indicate the segments selected by each method, and the two rows of edged boxes within each method indicate the two distinct true annotations. The results clearly demonstrate the effectiveness of SummDiff in capturing multiple plausible summaries for a video. See another example in \ref{['appendix:qual']}.