Shot Sequence Ordering for Video Editing: Benchmarks, Metrics, and Cinematology-Inspired Computing Methods

TL;DR

This work addresses the lack of public benchmarks for Shot Sequence Ordering (SSO) in AI-assisted video editing by introducing AVE-Order and ActivityNet-Order, two datasets with complete shot sequences and labels. It advances SSO evaluation by adopting Kendall Tau distance and proposes a Kendall Tau Distance-Cross Entropy Loss (KTD-CE) that blends classification and ordering signals. The authors further introduce Cinematology Embedding to inject film-grammars through movie metadata and shot-type priors via a Video Transformer framework, and create AVE-Meta for metadata linkage. Experimental results on both benchmarks demonstrate that KTD-CE and Cinematology Embedding substantially improve SSO accuracy, and ablations confirm the contributions of each component. Overall, the work provides public datasets, a principled ordering metric, a specialized loss, and metadata-informed modeling to push forward professional-level SSO in AI-assisted video editing.

Abstract

With the rising popularity of short video platforms, the demand for video production has increased substantially. However, high-quality video creation continues to rely heavily on professional editing skills and a nuanced understanding of visual language. To address this challenge, the Shot Sequence Ordering (SSO) task in AI-assisted video editing has emerged as a pivotal approach for enhancing video storytelling and the overall viewing experience. Nevertheless, the progress in this field has been impeded by a lack of publicly available benchmark datasets. In response, this paper introduces two novel benchmark datasets, AVE-Order and ActivityNet-Order. Additionally, we employ the Kendall Tau distance as an evaluation metric for the SSO task and propose the Kendall Tau Distance-Cross Entropy Loss. We further introduce the concept of Cinematology Embedding, which incorporates movie metadata and shot labels as prior knowledge into the SSO model, and constructs the AVE-Meta dataset to validate the method's effectiveness. Experimental results indicate that the proposed loss function and method substantially enhance SSO task accuracy. All datasets are publicly accessible at https://github.com/litchiar/ShotSeqBench.

Figures (6)

• Figure 1: (a) Workflow of professional video editors in traditional editing, involving manual arrangement of unordered shot clips to construct a coherent, narrative-rich video. (b) Application of AI-assisted systems in video editing, where AI analyzes shot content and intelligently arranges them, enabling users to generate a coherent shot sequence efficiently.
• Figure 2: The label distribution of AVE-Order and ActivityNet-Order.
• Figure 3: Kendall Tau distance matrix when $k=3$
• Figure 4: Distribution of shot type frequencies across different film genres.
• Figure 5: Shot Label Prediction Methods.(a) Directly using labels as text embeddings for CLIP (b) Using prompt-optimized labels as text embeddings for CLIP (c) Predicting shot labels using our proposed UQ-Net cite15.