Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward

Yolo Yunlong Tang, Siting Xu, Teng Wang, Qin Lin, Qinglin Lu, Feng Zheng

TL;DR

This work addresses automatic ad video editing by framing segment assemblage as a constrained combinatorial problem and delivering an end-to-end solution called Multi-modal Segment Assemblage Network (M-SAN). M-SAN fuses visual, audio, and text features through a multi-modal representation and uses an Encoder-Decoder Ptr-Net with attention to select and order segments under duration constraints, trained with an Importance-Coherence Reward via policy gradient. The authors introduce the Ads-1k dataset and a unified Imp-Coh@Time metric to evaluate the trade-off between information importance, narrative coherence, and output duration. Experimental results show state-of-the-art performance on Imp-Coh@T with ablations confirming the benefit of multi-modal fusion and the proposed reward design. This work advances practical, coherent, and efficient automatic ad video editing with potential impact on scalable ad content generation.

Abstract

Advertisement video editing aims to automatically edit advertising videos into shorter videos while retaining coherent content and crucial information conveyed by advertisers. It mainly contains two stages: video segmentation and segment assemblage. The existing method performs well at video segmentation stages but suffers from the problems of dependencies on extra cumbersome models and poor performance at the segment assemblage stage. To address these problems, we propose M-SAN (Multi-modal Segment Assemblage Network) which can perform efficient and coherent segment assemblage task end-to-end. It utilizes multi-modal representation extracted from the segments and follows the Encoder-Decoder Ptr-Net framework with the Attention mechanism. Importance-coherence reward is designed for training M-SAN. We experiment on the Ads-1k dataset with 1000+ videos under rich ad scenarios collected from advertisers. To evaluate the methods, we propose a unified metric, Imp-Coh@Time, which comprehensively assesses the importance, coherence, and duration of the outputs at the same time. Experimental results show that our method achieves better performance than random selection and the previous method on the metric. Ablation experiments further verify that multi-modal representation and importance-coherence reward significantly improve the performance. Ads-1k dataset is available at: https://github.com/yunlong10/Ads-1k

Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward

TL;DR

This work addresses automatic ad video editing by framing segment assemblage as a constrained combinatorial problem and delivering an end-to-end solution called Multi-modal Segment Assemblage Network (M-SAN). M-SAN fuses visual, audio, and text features through a multi-modal representation and uses an Encoder-Decoder Ptr-Net with attention to select and order segments under duration constraints, trained with an Importance-Coherence Reward via policy gradient. The authors introduce the Ads-1k dataset and a unified Imp-Coh@Time metric to evaluate the trade-off between information importance, narrative coherence, and output duration. Experimental results show state-of-the-art performance on Imp-Coh@T with ablations confirming the benefit of multi-modal fusion and the proposed reward design. This work advances practical, coherent, and efficient automatic ad video editing with potential impact on scalable ad content generation.

Abstract

Advertisement video editing aims to automatically edit advertising videos into shorter videos while retaining coherent content and crucial information conveyed by advertisers. It mainly contains two stages: video segmentation and segment assemblage. The existing method performs well at video segmentation stages but suffers from the problems of dependencies on extra cumbersome models and poor performance at the segment assemblage stage. To address these problems, we propose M-SAN (Multi-modal Segment Assemblage Network) which can perform efficient and coherent segment assemblage task end-to-end. It utilizes multi-modal representation extracted from the segments and follows the Encoder-Decoder Ptr-Net framework with the Attention mechanism. Importance-coherence reward is designed for training M-SAN. We experiment on the Ads-1k dataset with 1000+ videos under rich ad scenarios collected from advertisers. To evaluate the methods, we propose a unified metric, Imp-Coh@Time, which comprehensively assesses the importance, coherence, and duration of the outputs at the same time. Experimental results show that our method achieves better performance than random selection and the previous method on the metric. Ablation experiments further verify that multi-modal representation and importance-coherence reward significantly improve the performance. Ads-1k dataset is available at: https://github.com/yunlong10/Ads-1k
Paper Structure (40 sections, 18 equations, 7 figures, 6 tables)

This paper contains 40 sections, 18 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Editing
  4. Text Coherence Prediction and Evaluation
  5. Neural Combinatorial Optimization
  6. Method
  7. Problem Formulation
  8. Architecture
  9. Video Segmentation Module.
  10. Multi-modal Representation Extraction Module.
  11. Assemblage Module.
  12. Reward Design
  13. Importance Reward.
  14. Coherence Reward.
  15. Importance-Coherence Reward.
  16. ...and 25 more sections

Figures (7)

  • Figure 1: The two stages of ad video editing: video segmentation and segment assemblage.
  • Figure 2: The Architecture of M-SAN
  • Figure 3: PPL map and coherence reward
  • Figure 4: Comparison results with $\beta=0.0/0.3/0.5/0.7$.
  • Figure 5: Visualization. Source video and videos assembled by SAM and M-SAN given target duration $T=15s$. $\tau$ is the actual duration of result.
  • ...and 2 more figures