Finding the Right Moment: Human-Assisted Trailer Creation via Task Composition

TL;DR

This work introduces Graph-Trailer, a multimodal, graph-based framework for automatically identifying trailer-worthy moments in full-length films by jointly modeling narrative structure (turning points) and sentiment flow. It leverages a contrastive knowledge-distillation regime where an auxiliary text network trained on screenplays distills privileged information into a video-based main network, with both prediction and representation consistency losses guiding joint training. An interpretable graph-traversal algorithm selects trailer shot sequences, and an interactive tool enables humans to review and refine proposals, reducing manual effort to under 30 minutes while delivering results competitive with expert selections. The approach is evaluated on full-length movies with a held-out trailer dataset, showing improvements over baselines in both automatic trailer moment identification and human-in-the-loop trailer creation, and revealing valuable insights into turning points, sentiment flow, and cross-modal integration for trailer design.

Abstract

Movie trailers perform multiple functions: they introduce viewers to the story, convey the mood and artistic style of the film, and encourage audiences to see the movie. These diverse functions make trailer creation a challenging endeavor. In this work, we focus on finding trailer moments in a movie, i.e., shots that could be potentially included in a trailer. We decompose this task into two subtasks: narrative structure identification and sentiment prediction. We model movies as graphs, where nodes are shots and edges denote semantic relations between them. We learn these relations using joint contrastive training which distills rich textual information (e.g., characters, actions, situations) from screenplays. An unsupervised algorithm then traverses the graph and selects trailer moments from the movie that human judges prefer to ones selected by competitive supervised approaches. A main advantage of our algorithm is that it uses interpretable criteria, which allows us to deploy it in an interactive tool for trailer creation with a human in the loop. Our tool allows users to select trailer shots in under 30 minutes that are superior to fully automatic methods and comparable to (exclusive) manual selection by experts.

Figures (11)

• Figure 1: Turning points and their definitions Hauge:2017.
• Figure 2: Graph-Trailer performs walks (bold line) in a movie graph to generate proposal trailer sequences. Nodes in the graph are shots and edges denote relations between them. Each shot is characterized by a sentiment score (green/red shades for positive/negative values) and labels describing important events (thick circles).
• Figure 3: Two networks process different views of the movie with different degrees of granularity. Our main network (right side) takes as input multimodal fine-grained shot representations based on the movie's video stream. The auxiliary text-based network (left side) processes textual scene representations which are coarse-grained and based on the movie's screenplay. The networks are trained jointly on TP identification with losses enforcing prediction and representation consistency between them.
• Figure 4: Run of Graph-Trailer algorithm for the movie "The Shining". Step 1 illustrates the shot-level graph (pruned for better visualization) with colored nodes representing the different types of TPs predicted in the movie (i.e., TP1, TP2, TP3, TP4, TP5). Our algorithm starts by sampling a shot identified as TP1 (Step 1). For each next step, we only consider the immediate neighborhood of the current shot (i.e., 6--12 neighbors) and select the next shot based on the following criteria: (1) semantic similarity, (2) time proximity, (3) narrative structure, and (4) sentiment intensity (Steps 2--5 or beyond). Finally, we assemble the proposal trailer (Final step) by concatenating the shots in the path. When our algorithm is used as an interactive tool, it allows users to review candidate shots at each step and manually select the best one while taking into account our criteria. Users create trailers by only reviewing around 10% of the movie.
• Figure 5: Users can manually review a limited set of shots to be included in the trailer given metadata, such as importance, sentiment intensity, and transition match; they create trailers interactively step by step by selecting and (optionally) trimming best shots.