CFSum: A Transformer-Based Multi-Modal Video Summarization Framework With Coarse-Fine Fusion

TL;DR

CFSum tackles multi-modal video summarization by explicitly integrating video, text, and audio through a two-stage fusion pipeline. A modal autoencoder strengthens intra-modal features, followed by a coarse-grained fusion and subsequent fine-grained cross-modal interactions to produce a unified representation for saliency prediction. Empirical results on TVSum, YouTube Highlights, and QVHighlights show state-of-the-art performance with multi-modal inputs, and ablations confirm the contribution of each modality and module. The approach offers a practical, parameter-efficient path to audio-augmented MMVS with broad applicability across diverse video content.

Abstract

Video summarization, by selecting the most informative and/or user-relevant parts of original videos to create concise summary videos, has high research value and consumer demand in today's video proliferation era. Multi-modal video summarization that accomodates user input has become a research hotspot. However, current multi-modal video summarization methods suffer from two limitations. First, existing methods inadequately fuse information from different modalities and cannot effectively utilize modality-unique features. Second, most multi-modal methods focus on video and text modalities, neglecting the audio modality, despite the fact that audio information can be very useful in certain types of videos. In this paper we propose CFSum, a transformer-based multi-modal video summarization framework with coarse-fine fusion. CFSum exploits video, text, and audio modal features as input, and incorporates a two-stage transformer-based feature fusion framework to fully utilize modality-unique information. In the first stage, multi-modal features are fused simultaneously to perform initial coarse-grained feature fusion, then, in the second stage, video and audio features are explicitly attended with the text representation yielding more fine-grained information interaction. The CFSum architecture gives equal importance to each modality, ensuring that each modal feature interacts deeply with the other modalities. Our extensive comparative experiments against prior methods and ablation studies on various datasets confirm the effectiveness and superiority of CFSum.

Figures (2)

• Figure 1: Given video, audio, and text as inputs, CFSum leverages multi-modal coarse-fine fusion to generate a summary video by selecting from the original video the most important parts or those relevant to the user's input.
• Figure 2: CFSum model architecture. Upon receiving multi-modal inputs, CFSum first leverages modal autoencoders to enhance features by intra-modal learning. The results are then input to a modal fusion module that performs coarse-grained fusion. This is followed by feature interaction modules that perform fine-grained fusion. Finally, a saliency prediction head predicts the saliency scores.