Your Interest, Your Summaries: Query-Focused Long Video Summarization

TL;DR

This paper tackles query-focused video summarization for long videos by introducing FCSNA-QFVS, a Fully Convolutional Sequence Network augmented with Local Self-Attention, Query-Guided Segment-Level Attention, and Global Attention. The architecture comprises a feature learning module based on eight 1D temporal convolution blocks, a three-attention fusion stage, a deconvolution layer to restore temporal length, and a shot scoring module that predicts per-shot relevance to a user query, selecting the top 2% shots for the final summary. Evaluated on a benchmark egocentric QFVS dataset, the method achieves higher F1 scores than prior approaches and provides qualitative analyses illustrating query relevance and shot selection. Overall, the approach enables efficient, parallelizable processing of long videos with improved alignment to user queries, enhancing the usefulness of automated summaries in real-world applications.

Abstract

Generating a concise and informative video summary from a long video is important, yet subjective due to varying scene importance. Users' ability to specify scene importance through text queries enhances the relevance of such summaries. This paper introduces an approach for query-focused video summarization, aiming to align video summaries closely with user queries. To this end, we propose the Fully Convolutional Sequence Network with Attention (FCSNA-QFVS), a novel approach designed for this task. Leveraging temporal convolutional and attention mechanisms, our model effectively extracts and highlights relevant content based on user-specified queries. Experimental validation on a benchmark dataset for query-focused video summarization demonstrates the effectiveness of our approach.

Figures (5)

• Figure 1: Illustration of the Difference Between Generic Video Summarizers and Query-Focused Video Summarizers for a Given Long Video and Query.
• Figure 2: Overview of FCSNA-QFVS. Given a long video and a text query as input, we first divide the video into non-overlapping shots and group them into non-overlapping segments. Next, we pass the segmented video features to the feature learning module, where we learn visual features using eight sequential convolutional blocks. We then process these learned visual features through Local Self-Attention (LSA), Query-Guided Segment Attention (QGSA), and Global Attention (GA) to obtain locally important and globally query-guided features. We restore the original temporal length using two sequential deconvolutional layers. The feature learning network outputs the learned shot features, which we then pass to the shot scoring module to obtain a query relevance score for each shot. Finally, we generate the query-focused video summary based on these shot scores.
• Figure 3: Illustration of Local Self-Attention (LSA): Finding local importance among shots within each segment for all segments.
• Figure 4: First, we extract query-guided shot features. Then, we aggregate these features to obtain query-guided segment-level attentive features. Next, we determine the global relationship among segments by evaluating the relationship between the query-guided segment-level features and each shot within a segment across all segments using global attention.
• Figure 5: Illustration of our qualitative results for queries 'Food' and 'Sky'