Self-Paced and Self-Corrective Masked Prediction for Movie Trailer Generation

TL;DR

This work tackles automatic movie trailer generation by addressing error propagation in existing pipelines. It proposes SSMP, a self-paced masked-prediction framework that enables bi-directional contextual modeling and progressive self-correction via a Transformer encoder trained to reconstruct masked trailer shots from a movie-shot prompt. A self-paced mask ratio scheduler adapts task difficulty during training, improving efficiency and stability, while a self-correction generation process revisits decisions to mimic human editors. Experiments on an extended trailer dataset show SSMP achieving state-of-the-art results against selection-then-ranking and autoregressive baselines, with strong generalization to newly released movies and corroborating user studies. Post-processing steps further integrate narrations and music to yield cohesive, publishable trailers.

Abstract

As a challenging video editing task, movie trailer generation involves selecting and reorganizing movie shots to create engaging trailers. Currently, most existing automatic trailer generation methods employ a "selection-then-ranking" paradigm (i.e., first selecting key shots and then ranking them), which suffers from inevitable error propagation and limits the quality of the generated trailers. Beyond this paradigm, we propose a new self-paced and self-corrective masked prediction method called SSMP, which achieves state-of-the-art results in automatic trailer generation via bi-directional contextual modeling and progressive self-correction. In particular, SSMP trains a Transformer encoder that takes the movie shot sequences as prompts and generates corresponding trailer shot sequences accordingly. The model is trained via masked prediction, reconstructing each trailer shot sequence from its randomly masked counterpart. The mask ratio is self-paced, allowing the task difficulty to adapt to the model and thereby improving model performance. When generating a movie trailer, the model fills the shot positions with high confidence at each step and re-masks the remaining positions for the next prediction, forming a progressive self-correction mechanism that is analogous to how human editors work. Both quantitative results and user studies demonstrate the superiority of SSMP in comparison to existing automatic movie trailer generation methods. Demo is available at: https://github.com/Dixin-Lab/SSMP.

Figures (5)

• Figure 1: Illustrations of different trailer generation paradigms. Both classic selection-then-ranking and recent auto-regressive paradigms suffer from error propagation. Our method mitigates this issue, modeling bi-directional context and achieving self-correction via progressive re-masking. The shots used in this figure are from the movie Titanic.
• Figure 2: An illustration of our SSMP method. (a) shows the training pipeline, including the self-paced masking and probability computation. (b) shows the generation pipeline, including iterative masked prediction and re-masking of partially masked positions.
• Figure 3: (a) Red boxes mark incorrect predictions, and blue boxes highlight the predicted and real positions of a correctly selected shot. (b) Radar chart illustrating the average scores of different methods across four aspects.
• Figure 4: Mask ratio and prediction accuracy over optimization steps. Ratios in (b) are averaged over every 100 steps.
• Figure 5: Mask ratio over training steps for different settings.