Reframe Anything: LLM Agent for Open World Video Reframing

TL;DR

Reframe Any Video Agent (RAVA), a LLM-based agent that leverages visual foundation models and human instructions to restructure visual content for video reframing, is introduced, demonstrating its potential as a tool for AI-powered video editing.

Abstract

The proliferation of mobile devices and social media has revolutionized content dissemination, with short-form video becoming increasingly prevalent. This shift has introduced the challenge of video reframing to fit various screen aspect ratios, a process that highlights the most compelling parts of a video. Traditionally, video reframing is a manual, time-consuming task requiring professional expertise, which incurs high production costs. A potential solution is to adopt some machine learning models, such as video salient object detection, to automate the process. However, these methods often lack generalizability due to their reliance on specific training data. The advent of powerful large language models (LLMs) open new avenues for AI capabilities. Building on this, we introduce Reframe Any Video Agent (RAVA), a LLM-based agent that leverages visual foundation models and human instructions to restructure visual content for video reframing. RAVA operates in three stages: perception, where it interprets user instructions and video content; planning, where it determines aspect ratios and reframing strategies; and execution, where it invokes the editing tools to produce the final video. Our experiments validate the effectiveness of RAVA in video salient object detection and real-world reframing tasks, demonstrating its potential as a tool for AI-powered video editing.

Figures (6)

• Figure 1: The overview of open world video reframing task. Even within the same video, different viewers may focus on different subjects of interest. Therefore, it is essential to implement video reframing based on user instructions to achieve specific objectives.
• Figure 2: The overall workflow of our proposed Reframe Any Video Agent (RAVA). RAVA is capable of receiving user dialogue inputs with a language user interface (LUI) tailored for reframing tasks, and invokes the grounding function to retrieve object information within the video, then reframes the video automatically following the request from the users.
• Figure 3: The video editing tools in RAVA are described as follows: The first line is about 'Layout settings', where 'L' determines the number of selected objects in the video. The second line is 'Effect In-scene', represents the visual effects within a scene, divided into 'Zoom in' and 'Zoom out'. The third line is 'Effect Trans-scene', indicates the visual effects for scene transitions, divided into 'Fade in' and 'Fade out'.
• Figure 4: The qualitative results on two video salient object detection datasets in comparison with two state-of-the-art methods. The findings indicate that RAVA copes well even in the presence of occlusions and distractions, and at times, it even surpasses the results of human annotations.
• Figure 5: The cases where using RAVA might fail include situations (a) and (b), where it struggles to effectively handle composite objects and clusters of closely situated objects. However, in cases (c) and (d), despite the differences in perception of salient objects, we consider the segmentation results to be acceptable.