Modality-Aware Shot Relating and Comparing for Video Scene Detection

TL;DR

The paper tackles video scene detection by introducing MASRC, a modality-aware framework that separately models long-range entity relations via an Entity Jumping Graph and short-range place relations via a Place Continuity Graph. It then enhances ending-shot detection through Multi-shots Comparison Detection, which encodes context changes by comparing fore-and-aft shot semantics. MASRC is trainable in self-supervised, supervised, or transfer settings and demonstrates state-of-the-art performance on MovieNet, OVSD, and BBC with strong generalization and efficiency. The work contributes a modular, multimodal approach that leverages graph-based relational reasoning and contextual comparison to improve scene boundary detection in real-world video data.

Abstract

Video scene detection involves assessing whether each shot and its surroundings belong to the same scene. Achieving this requires meticulously correlating multi-modal cues, $\it{e.g.}$ visual entity and place modalities, among shots and comparing semantic changes around each shot. However, most methods treat multi-modal semantics equally and do not examine contextual differences between the two sides of a shot, leading to sub-optimal detection performance. In this paper, we propose the $\bf{M}$odality-$\bf{A}$ware $\bf{S}$hot $\bf{R}$elating and $\bf{C}$omparing approach (MASRC), which enables relating shots per their own characteristics of visual entity and place modalities, as well as comparing multi-shots similarities to have scene changes explicitly encoded. Specifically, to fully harness the potential of visual entity and place modalities in modeling shot relations, we mine long-term shot correlations from entity semantics while simultaneously revealing short-term shot correlations from place semantics. In this way, we can learn distinctive shot features that consolidate coherence within scenes and amplify distinguishability across scenes. Once equipped with distinctive shot features, we further encode the relations between preceding and succeeding shots of each target shot by similarity convolution, aiding in the identification of scene ending shots. We validate the broad applicability of the proposed components in MASRC. Extensive experimental results on public benchmark datasets demonstrate that the proposed MASRC significantly advances video scene detection.

Figures (15)

• Figure 1: Intuition behind our proposed modality-aware shot relating and comparing. (a) Relating: We capture long-term shot relations by linking intermittently appearing identical entities and short-term shot relations by relating consecutive shots depicting the same place. (b) Comparing: Consistency drawn from comparisons between fore-and-aft contexts is an indicator to classify whether a target shot is an ending shot.
• Figure 2: Diagram of how the proposed MASRC determines whether a target shot (in the red border) is an ending shot or not. We build an entity jumping graph and a place continuity graph for GCN message passing to separately embed long and short shot relations into shot representations. Relying on similarity comparison between the fore-and-aft shots of each target shot as well as further similarity change encoding by convolution, the probability of target shots being ending shots can be better predicted by a simple MLP classifier.
• Figure 3: Each wide shot depicts an overview of a place, and its detail shots zoom in on specific details within the same place.
• Figure 4: Identification of wide shots and detail shots.
• Figure 5: Comparison between our MASR and HR msd_mhrt under the same detector, AD msd_mhrt or MLP msd_tran4fer.