Multimodal Fusion and Coherence Modeling for Video Topic Segmentation

TL;DR

Addresses video topic segmentation (VTS) by leveraging multimodal cues (visual and textual) and coherence modeling to surpass unsupervised baselines and strengthen supervised VTS. Proposes MMVTS, a supervised multimodal sequence labeling model with middle fusion across modalities and a suite of fusion architectures based on Cross-Attention and Mixture-of-Experts, complemented by multimodal contrastive pre-training. Introduces a KDE-based, task-tailored pre-training objective and two fine-tuning losses for multimodal coherence ($l_{cma}$, $l_{mcssl}$) to align modalities and emphasize topic-level coherence, and releases CLVTS, a large-scale Chinese educational video dataset. Experiments on AVLecture and CLVTS demonstrate state-of-the-art performance and validate the effectiveness of pre-training, MFL choices, and coherence modeling for educational video understanding.

Abstract

The video topic segmentation (VTS) task segments videos into intelligible, non-overlapping topics, facilitating efficient comprehension of video content and quick access to specific content. VTS is also critical to various downstream video understanding tasks. Traditional VTS methods using shallow features or unsupervised approaches struggle to accurately discern the nuances of topical transitions. Recently, supervised approaches have achieved superior performance on video action or scene segmentation over unsupervised approaches. In this work, we improve supervised VTS by thoroughly exploring multimodal fusion and multimodal coherence modeling. Specifically, (1) we enhance multimodal fusion by exploring different architectures using cross-attention and mixture of experts. (2) To generally strengthen multimodality alignment and fusion, we pre-train and fine-tune the model with multimodal contrastive learning. (3) We propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task for enhancing multimodal coherence modeling for VTS. We evaluate the proposed approaches on educational videos, in the form of lectures, due to the vital role of topic segmentation of educational videos in boosting learning experiences. Additionally, we introduce a large-scale Chinese lecture video dataset to augment the existing English corpus, promoting further research in VTS. Experiments on both English and Chinese lecture datasets demonstrate that our model achieves superior VTS performance compared to competitive unsupervised and supervised baselines.

Figures (10)

• Figure 1: Overall Model Architecture
• Figure 2: Merge-Attention
• Figure 3: Co-Attention
• Figure 4: Merge-Attention with MoE
• Figure 5: Co-Attention with MoE