From Shots to Stories: LLM-Assisted Video Editing with Unified Language Representations

TL;DR

This work introduces LLM-Assisted Video Editing, a framework that uses L-Storyboard to translate video shots into a unified language representation for LLM reasoning, coupled with StoryFlow to stabilize divergent task outputs. The authors categorize editing tasks into Convergent and Divergent, achieving strong results on Shot Attributes Classification and Next Shot Selection, while significantly improving Shot Sequence Ordering through the two-phase Divergent-then-Convergent strategy. The approach emphasizes interpretability and privacy by keeping processing local and bypassing heavy cloud-based pipelines. Empirical results on AVE and ActivityNet100 demonstrate competitive metrics across tasks, with StoryFlow yielding higher coherence and stability in sequencing. Overall, the work offers a scalable, prompt-driven framework for intelligent video editing that can adapt to new tasks with minimal architectural changes.

Abstract

Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated remarkable reasoning and generalization capabilities in video understanding; however, their application in video editing remains largely underexplored. This paper presents the first systematic study of LLMs in the context of video editing. To bridge the gap between visual information and language-based reasoning, we introduce L-Storyboard, an intermediate representation that transforms discrete video shots into structured language descriptions suitable for LLM processing. We categorize video editing tasks into Convergent Tasks and Divergent Tasks, focusing on three core tasks: Shot Attributes Classification, Next Shot Selection, and Shot Sequence Ordering. To address the inherent instability of divergent task outputs, we propose the StoryFlow strategy, which converts the divergent multi-path reasoning process into a convergent selection mechanism, effectively enhancing task accuracy and logical coherence. Experimental results demonstrate that L-Storyboard facilitates a more robust mapping between visual information and language descriptions, significantly improving the interpretability and privacy protection of video editing tasks. Furthermore, StoryFlow enhances the logical consistency and output stability in Shot Sequence Ordering, underscoring the substantial potential of LLMs in intelligent video editing.

Figures (8)

• Figure 1: Overview of our proposed L-Storyboard and AI-assisted video editing tasks. (a) L-Storyboard (b) Shot Attributes Classification (c) Next Shot Selection (d) Shot Sequence Ordering
• Figure 2: The prompt design adopted for describing video content.
• Figure 3: A complete L-Storyboard example, derived from Figure \ref{['fig:1']} (c).
• Figure 4: The prompt design used in Shot Attributes Classification.
• Figure 5: The prompt design used in Next Shot Selection.