Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarization

TL;DR

This work tackles zero-shot video summarization by integrating dataset-adaptive rubrics with pseudo-labeled, prompt-driven reasoning in large language models. A two-stage pipeline builds pseudo labels from limited ground-truth data to craft rubric blocks and prompts, which are then used to score scenes with contextual information to preserve narrative coherence. Experiments on SumMe, TVSum, and QFVS demonstrate stable, competitive performance and clear gains over a strong zero-shot baseline, without any supervised training. The approach emphasizes interpretability, training-free generalization, and robustness to dataset shifts, offering a scalable path for both generic and query-focused video summarization across diverse domains.

Abstract

We propose a rubric-guided, pseudo-labeled, and prompt-driven zero-shot video summarization framework that bridges large language models with structured semantic reasoning. A small subset of human annotations is converted into high-confidence pseudo labels and organized into dataset-adaptive rubrics defining clear evaluation dimensions such as thematic relevance, action detail, and narrative progression. During inference, boundary scenes, including the opening and closing segments, are scored independently based on their own descriptions, while intermediate scenes incorporate concise summaries of adjacent segments to assess narrative continuity and redundancy. This design enables the language model to balance local salience with global coherence without any parameter tuning. Across three benchmarks, the proposed method achieves stable and competitive results, with F1 scores of 57.58 on SumMe, 63.05 on TVSum, and 53.79 on QFVS, surpassing zero-shot baselines by +0.85, +0.84, and +0.37, respectively. These outcomes demonstrate that rubric-guided pseudo labeling combined with contextual prompting effectively stabilizes LLM-based scoring and establishes a general, interpretable, and training-free paradigm for both generic and query-focused video summarization.

Figures (4)

• Figure 1: Illustrations of different methods.
• Figure 2: Overall architecture of the proposed framework.
• Figure 3: the case study presents the complete step-by-step summarization pipeline, including: (a) Initial Scene Boundaries detected from raw frame sequences, (b) Refined Scene Boundaries adjusted based on embedding similarity, (c) Scene Scores estimated by the language model using textual descriptions and user queries, (d) Normalized & Smoothed Scores obtained through temporal filtering, and (e) Frame-level Scores representing the final output aligned with averaged user annotations. Blue lines denote user annotation means, and yellow curves indicate model-predicted frame importance scores. This visualization highlights how the proposed framework progressively refines temporal segmentation and scoring to achieve coherent, human-aligned video summaries.
• Figure 4: Visualization of the selections made by our model, user summaries, and KTS segments.potapov2014category The blue bars represent human frame selections for each video, with each row indicating a distinct user summary. The yellow bars denote ground-truth fragments included in the final summary. The green bars show the frames selected by our model, and the red bars mark unselected frames. Representative video frames from each selected shot are included for intuitive understanding.