REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing

TL;DR

REGen addresses the challenge of producing coherent long-to-short video teasers that are grounded with exact quotable content from the source. It introduces a two-stage pipeline in which a finetuned LLM first generates a narrative with quotation placeholders, and a multimodal retriever fills those placeholders with quotable clips via a retrieval-augmented objective. The retriever operates in two variants, using text alone or text+visuals, and is trained with a multitask loss L = L_gen + αL_ret, selecting clips by maximizing a clip fitness CF = cos_sim(h, e_m). Evaluations on DocumentaryNet show improvements in coherence, alignment, and realism, with REGen-DQ excelling in direct quoting and REGen-IDQ-TV delivering strong grounding and narrative coherence in teaser generation. The work demonstrates a viable path toward factual, multimodal grounding in narrative generation for video editing, with potential extensions to additional modalities and domains.

Abstract

Short videos are an effective tool for promoting contents and improving knowledge accessibility. While existing extractive video summarization methods struggle to produce a coherent narrative, existing abstractive methods cannot `quote' from the input videos, i.e., inserting short video clips in their outputs. In this work, we explore novel video editing models for generating shorts that feature a coherent narrative with embedded video insertions extracted from a long input video. We propose a novel retrieval-embedded generation framework that allows a large language model to quote multimodal resources while maintaining a coherent narrative. Our proposed REGen system first generates the output story script with quote placeholders using a finetuned large language model, and then uses a novel retrieval model to replace the quote placeholders by selecting a video clip that best supports the narrative from a pool of candidate quotable video clips. We examine the proposed method on the task of documentary teaser generation, where short interview insertions are commonly used to support the narrative of a documentary. Our objective evaluations show that the proposed method can effectively insert short video clips while maintaining a coherent narrative. In a subjective survey, we show that our proposed method outperforms existing abstractive and extractive approaches in terms of coherence, alignment, and realism in teaser generation.

Figures (3)

• Figure 1: An overview of the proposed REGen system for long-to-short video editing. Given a long input video, we first transcribe the narrations and dialogues using a pretrained automatic speech recognition model, and then we use a finetuned large language model to generate the output story script with quote placeholders (i.e., the <QUOTE> token). For the generated narration, following xu2024teasergengeneratingteaserslong, we first synthesize the narration into audio using a text-to-speech synthesis model and apply a narration-visual matching algorithm to find accompanying visuals. For the generated quote placeholders, we propose an encoder-decoder based Quote Retriever to select a video clip that best supports the narrative from a pool of quotable video clips extracted from the input video. The proposed system represents a new hybrid video editing model that combines abstractive and extractive methods.
• Figure 2: An illustration of the proposed two-stage quote retriever REGen-IDQ. We finetune an encoder-decoder language model that learns to 1) fulfill quotation placeholders (i.e., the <QUOTE> token) and 2) produce an embedding vector $\mathbf{h}$ that summarizes the quotation content. We then use the embedding vector $\mathbf{h}$ as the query to retrieve a video clip from a pool of candidate quotable video clips extracted from the input video. The optimal clip is selected based on our proposed clip fitness measure (see \ref{['sec:retriever_module']} for its definition). In this work, we consider all non-narrator video clips as candidates for the quote retriever. Note that this framework can be generalized to support quoting materials in any modality such as audio and images as long as we can find a proper fitness measure.
• Figure 3: Comparison of infilling methods. The dotted lines indicate the median values.