Less is More: Label-Guided Summarization of Procedural and Instructional Videos

TL;DR

PRISM introduces a zero-shot, label-guided, context-aware framework for procedural and instructional video summarization. It combines adaptive frame sampling, semantic anchor generation, and temporally grounded aggregation to produce coherent, semantically rich summaries while processing under 5% of frames. Across YouCook2, ActivityNet Captions, TVSum, SumMe, and medical datasets, PRISM demonstrates strong semantic alignment (high METEOR and BERTScore) and competitive keyframe rankings, with notable efficiency gains. The approach highlights the value of bottom-up semantic anchoring and LLM-assisted synthesis for robust, domain-general video summarization, while acknowledging potential hallucinations and computational overhead as avenues for future improvement.

Abstract

Video summarization helps turn long videos into clear, concise representations that are easier to review, document, and analyze, especially in high-stakes domains like surgical training. Prior work has progressed from using basic visual features like color, motion, and structural changes to using pre-trained vision-language models that can better understand what's happening in the video (semantics) and capture temporal flow, resulting in more context-aware video summarization. We propose a three-stage framework, PRISM: Procedural Representation via Integrated Semantic and Multimodal analysis, that produces semantically grounded video summaries. PRISM combines adaptive visual sampling, label-driven keyframe anchoring, and contextual validation using a large language model (LLM). Our method ensures that selected frames reflect meaningful and procedural transitions while filtering out generic or hallucinated content, resulting in contextually coherent summaries across both domain-specific and instructional videos. We evaluate our method on instructional and activity datasets, using reference summaries for instructional videos. Despite sampling fewer than 5% of the original frames, our summaries retain 84% semantic content while improving over baselines by as much as 33%. Our approach generalizes across procedural and domain-specific video tasks, achieving strong performance with both semantic alignment and precision.

Figures (6)

• Figure 1: PRISM visual pipeline: input video is semantically decomposed, filtered, and recombined into a coherent summary.
• Figure 2: A complete example of the workflow
• Figure 3: A Venn diagram illustrating overlap between frame embeddings and semantic labels where cosine similarity exceeds 0.9. Highlighted labels reflect procedural relevance across frames.
• Figure 4: Predicted and ground truth video captions with scores for a sample video from the ActivityNet Captions dataset.
• Figure 5: Predicted and ground truth video captions with scores for a sample video from the YouCook2 dataset.