HierSum: A Global and Local Attention Mechanism for Video Summarization

TL;DR

HierSum tackles instructional video summarization by fusing fine-grained subtitle cues with global task descriptions in a hierarchical, two-stage training framework. It aligns video and text through cross-modal attention and a parent-child protocol, and exploits YouTube most-replayed statistics as an additional supervisory signal. Across TVSum, BLiSS, Mr.HiSum, and WikiHow, HierSum achieves improved rank correlations and cosine-based similarities, with notable gains when pre-trained on replay statistics and when leveraging both local and global text cues. The approach demonstrates robust cross-dataset transfer and provides a scalable dataset collection pipeline for instructional content. Overall, HierSum advances multi-modal hierarchical learning for task-focused video summarization with practical impact on search, retrieval, and educational video understanding.

Abstract

Video summarization creates an abridged version (i.e., a summary) that provides a quick overview of the video while retaining pertinent information. In this work, we focus on summarizing instructional videos and propose a method for breaking down a video into meaningful segments, each corresponding to essential steps in the video. We propose \textbf{HierSum}, a hierarchical approach that integrates fine-grained local cues from subtitles with global contextual information provided by video-level instructions. Our approach utilizes the ``most replayed" statistic as a supervisory signal to identify critical segments, thereby improving the effectiveness of the summary. We evaluate on benchmark datasets such as TVSum, BLiSS, Mr.HiSum, and the WikiHow test set, and show that HierSum consistently outperforms existing methods in key metrics such as F1-score and rank correlation. We also curate a new multi-modal dataset using WikiHow and EHow videos and associated articles containing step-by-step instructions. Through extensive ablation studies, we demonstrate that training on this dataset significantly enhances summarization on the target datasets.

Figures (4)

• Figure 1: Instructional video on "How to Bake Salmon" from WikiHow showing ASR text and instructions from the associated WikiHow article. Subtitles sometimes miss important instructions such as "Leave the skin on", whereas global descriptions lack video-text alignment.
• Figure 2: Overview of HierSum: in the sub-clip level (child level) training, with $\mathcal{N}$ frames and $\mathcal{M}$ subtitles as input, the model predicts the important frames and sentences as summaries. During the parent-level training, the subtitles are replaced with global descriptions and the model is trained only to predict important video frames. Note that the model $\mathcal{F}$ is common and is trained in both stages. When training with most replayed scores, the classifier head predicts the scores for each frame.
• Figure 3: Performance comparison for different global and local training steps on Mr.HiSum. Global step=5 indicates that for every five samples of training with local subtitles, a sample with global description is trained. ASR only indicates that the model was trained only using the local subtitles.
• Figure 4: Qualitative Results. We show summaries from our method HierSum trained on Pseduo Summaries narasimhan2022tl and our dataset. HierSum, when trained on our dataset assigns higher scores to all the frames relevant, and lower scores that aren't crucial to the task, e.g WikiHow logo. We note a failure case in (c).