LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing

TL;DR

LAVE presents a plan-and-execute, LLM-powered agent for video editing that couples natural language interaction with language-augmented video descriptions to reduce editing barriers. The system provides five main LLM-powered functions (footage overview, idea brainstorming, video retrieval, storyboarding, and clip trimming) implemented atop automatically generated visual narrations, and offers both agent-assisted and manual editing modes. A user study with eight participants shows the approach improves accessibility and supports creativity, while highlighting variability in perceived usefulness and the need for preserving user agency and addressing potential biases. The work contributes a complete system, a computational pipeline for translating plans into executable actions, and design implications to guide future adaptive, human-centered agent-assisted editing tools with LLMs.

Abstract

Video creation has become increasingly popular, yet the expertise and effort required for editing often pose barriers to beginners. In this paper, we explore the integration of large language models (LLMs) into the video editing workflow to reduce these barriers. Our design vision is embodied in LAVE, a novel system that provides LLM-powered agent assistance and language-augmented editing features. LAVE automatically generates language descriptions for the user's footage, serving as the foundation for enabling the LLM to process videos and assist in editing tasks. When the user provides editing objectives, the agent plans and executes relevant actions to fulfill them. Moreover, LAVE allows users to edit videos through either the agent or direct UI manipulation, providing flexibility and enabling manual refinement of agent actions. Our user study, which included eight participants ranging from novices to proficient editors, demonstrated LAVE's effectiveness. The results also shed light on user perceptions of the proposed LLM-assisted editing paradigm and its impact on users' creativity and sense of co-creation. Based on these findings, we propose design implications to inform the future development of agent-assisted content editing.

Figures (8)

• Figure 1: LAVE's video editing timeline: Users can drag and drop video clips to rearrange their order. The order can also be changed through LAVE's video editing agent's storyboarding function. To trim a clip, users can double-click it, revealing a pop-up window for trimming as shown in Figure \ref{['fig:trimming']} .
• Figure 2: LAVE's language-augmented video gallery features each video with a semantic title and its length (A). When users hover their cursor over a video, a detailed summary appears, allowing them to preview the video content without playing it (B). Users can select multiple videos to add to the timeline. Selected videos will be highlighted in light grey (C) and those already added will appear with faded opacity (D).
• Figure 3: LAVE's clip-trimming window displays user guide (A) and video frames sampled every second from the clip (B). Users can manually set the start and end frames for trimming. Alternatively, they can use the LLM-powered trimming feature with commands like "Give me 5 seconds focusing on the nearby cherry blossom tree." (D). With this approach, the trim automatically adjusts and includes a rationale explaining the LLM's choice (C). Frames not included in the trimmed clip are displayed in a dimmed color.
• Figure 4: LAVE's video editing agent operates in two states: Planning and Executing. In the Planning state (left), users provide editing commands (A). The agent then clarifies the goal (B) and proposes actionable steps to achieve the goal (C). Users have the option to revise the plan if they are not satisfied with the proposed steps. Upon user approval of the plan, the agent transitions to the Executing state (right). In this state, the user approves the agent's actions sequentially. Following each action, the agent presents the results (Ds). If additional actions are outlined in the plan, the agent notifies the user of the next action (Es) and waits for their approval (Fs).
• Figure 5: LAVE's plan-and-execute agent design: Upon receiving an input containing the user's editing command, a planning prompt is constructed. This prompt includes the planning instruction, past conversations, and the new user command. It is then sent to the LLM to produce an action plan, which reflects the user's editing goal and outlines actions to assist the user in achieving this goal. Each action is accompanied by a context, which provides additional information relevant to the action, such as a language query for video retrieval. The user reviews and approves the actions one by one. After an action is approved, it is translated into actual Python function calls and executed. This process continues for all the actions in the plan, unless the user decides to provide new instructions to revise or cancel the plan.