Personalized Video Summarization using Text-Based Queries and Conditional Modeling

TL;DR

This work addresses the challenge of efficiently navigating long video content by enabling personalized, text-guided video summarization. It advances the field through four interrelated strands: (1) query-dependent summaries using a multi-modal end-to-end model that fuses text and visual data, (2) GPT-2–based contextualized encoding and attention mechanisms for improved cross-modal understanding, (3) a conditional modeling framework that explicitly accounts for human-like non-visual factors and introduces helper distributions and a conditional attention module for explainability, and (4) a pseudo-label supervision strategy with segment-level pretext tasks and a semantics booster to mitigate data scarcity. Across multiple benchmarks (TVSum, SumMe, QueryVS), the methods achieve state-of-the-art performance in accuracy and F1-score, demonstrating improved control, interpretability, and robustness. The practical impact lies in more efficient video exploration, personalized content curation, and scalable evaluation pipelines for query-based video summarization.

Abstract

The proliferation of video content on platforms like YouTube and Vimeo presents significant challenges in efficiently locating relevant information. Automatic video summarization aims to address this by extracting and presenting key content in a condensed form. This thesis explores enhancing video summarization by integrating text-based queries and conditional modeling to tailor summaries to user needs. Traditional methods often produce fixed summaries that may not align with individual requirements. To overcome this, we propose a multi-modal deep learning approach that incorporates both textual queries and visual information, fusing them at different levels of the model architecture. Evaluation metrics such as accuracy and F1-score assess the quality of the generated summaries. The thesis also investigates improving text-based query representations using contextualized word embeddings and specialized attention networks. This enhances the semantic understanding of queries, leading to better video summaries. To emulate human-like summarization, which accounts for both visual coherence and abstract factors like storyline consistency, we introduce a conditional modeling approach. This method uses multiple random variables and joint distributions to capture key summarization components, resulting in more human-like and explainable summaries. Addressing data scarcity in fully supervised learning, the thesis proposes a segment-level pseudo-labeling approach. This self-supervised method generates additional data, improving model performance even with limited human-labeled datasets. In summary, this research aims to enhance automatic video summarization by incorporating text-based queries, improving query representations, introducing conditional modeling, and addressing data scarcity, thereby creating more effective and personalized video summaries.

Figures (23)

• Figure 1: The process of video summarization. Video summarization techniques strive to extract the most crucial and pertinent content from a video, condensing it into a concise representation. Traditional video summarization generates a summary independent of the context. To personalize the video summary, it can be created based on a given input query as context.
• Figure 2: An overview of the thesis.
• Figure 3: The concept of query-controllable video summarization. In query-controllable video summarization, a user provides a text-based query representing the desired video summary, directing the video summary generator to produce a summary aligned with the query. Notably, this method takes both the query and video as input. Consequently, when the same video input is paired with different queries, distinct query-dependent video summaries are generated. This differs from traditional video summarization, which relies solely on video input. For detailed information about the video summary controller and video summary generator, please refer to Figures \ref{['fig:controller_text_only']} and Figure \ref{['fig:generator_new']}.
• Figure 4: Illustrating the concept of our video summary controller. Within the controller, we gather all queries from our training set to construct a bag of words, subsequently forming a dictionary. This dictionary serves as the basis for embedding an input query.
• Figure 5: Illustration of the proposed video summary generator. The process begins by taking an input video and sampling it at a rate of $1$ fps. Subsequently, the sampled frames are fed into a CNN-based structure designed for extracting frame-based features. These features are then fused with the features derived from a text-based input query. Finally, the fused feature set is passed through a prediction layer to generate frame-based relevance scores. The predicted scores serve as the foundation for generating a query-dependent video summary.