VCoME: Verbal Video Composition with Multimodal Editing Effects

TL;DR

This work introduces verbal video composition with editing effects, a novel task that requires aligning multimodal editing cues across text, visuals, and audio with a narrative. The authors collect a large-scale dataset and propose VCoME, a segment-level multimodal framework that uses an LMM backbone to autoregressively determine trigger positions and corresponding editing effects, with prompt-based controls for density and style. Through extensive experiments, including ablations and a user study, VCoME achieves high-quality compositions and demonstrates professional aesthetics with about 85x editing efficiency compared to human editors. The work advances accessible, automated video editing for non-professionals and lays groundwork for future expansion into additional editing elements and richer user controls.

Abstract

Verbal videos, featuring voice-overs or text overlays, provide valuable content but present significant challenges in composition, especially when incorporating editing effects to enhance clarity and visual appeal. In this paper, we introduce the novel task of verbal video composition with editing effects. This task aims to generate coherent and visually appealing verbal videos by integrating multimodal editing effects across textual, visual, and audio categories. To achieve this, we curate a large-scale dataset of video effects compositions from publicly available sources. We then formulate this task as a generative problem, involving the identification of appropriate positions in the verbal content and the recommendation of editing effects for these positions. To address this task, we propose VCoME, a general framework that employs a large multimodal model to generate editing effects for video composition. Specifically, VCoME takes in the multimodal video context and autoregressively outputs where to apply effects within the verbal content and which effects are most appropriate for each position. VCoME also supports prompt-based control of composition density and style, providing substantial flexibility for diverse applications. Through extensive quantitative and qualitative evaluations, we clearly demonstrate the effectiveness of VCoME. A comprehensive user study shows that our method produces videos of professional quality while being 85$\times$ more efficient than professional editors.

Figures (11)

• Figure 1: Tasks in video editing, grouped by target modality. Some tasks generate visuals, while others provide audio, text, and additional elements. Our method generates compositions with multimodal effects for verbal videos. Comprehensive video editing should consider the composition of all these elements.
• Figure 2: Our framework. The input data comprises three modalities and is segmented by sentences. After being processed by encoders and projectors, the multimodal tokens are uniformly handled by the LMM. Subsequently, the composition is generated, detailing the arrangement of various effects on the original video. The final video is rendered according to this composition, utilizing the effect pool.
• Figure 3: Distribution of data length.
• Figure 4: Distribution of trigger positions.
• Figure 5: Distribution of stickers used.