Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM

TL;DR

Modular-Cam addresses the challenge of generating long, multi-scene videos with precise camera-view transitions by combining an LLM-driven scene decomposition (LLM-Director) with a modular motion-control framework (CamOperator) and cross-scene consistency mechanisms (AdaControlNet). A temporal-transformer-augmented base video generator provides intra-scene continuity, while LLM-guided selection composes motion modules to realize complex prompts; adaptive normalization and randomized blending ensure color and content consistency across scene boundaries. Empirical results on WebVid-10M show strong performance in motion smoothness and dynamic camera control, with competitive image quality and best cross-scene coherence, validated by ablations on AdaControlNet and LLM-Director. The approach offers scalable, plug-in motion modules and a director-like role for handling intricate user instructions, enabling practical, long-form, multi-scene video generation with fine-grained camera dynamics.

Abstract

Text-to-Video generation, which utilizes the provided text prompt to generate high-quality videos, has drawn increasing attention and achieved great success due to the development of diffusion models recently. Existing methods mainly rely on a pre-trained text encoder to capture the semantic information and perform cross attention with the encoded text prompt to guide the generation of video. However, when it comes to complex prompts that contain dynamic scenes and multiple camera-view transformations, these methods can not decompose the overall information into separate scenes, as well as fail to smoothly change scenes based on the corresponding camera-views. To solve these problems, we propose a novel method, i.e., Modular-Cam. Specifically, to better understand a given complex prompt, we utilize a large language model to analyze user instructions and decouple them into multiple scenes together with transition actions. To generate a video containing dynamic scenes that match the given camera-views, we incorporate the widely-used temporal transformer into the diffusion model to ensure continuity within a single scene and propose CamOperator, a modular network based module that well controls the camera movements. Moreover, we propose AdaControlNet, which utilizes ControlNet to ensure consistency across scenes and adaptively adjusts the color tone of the generated video. Extensive qualitative and quantitative experiments prove our proposed Modular-Cam's strong capability of generating multi-scene videos together with its ability to achieve fine-grained control of camera movements. Generated results are available at https://modular-cam.github.io.

Figures (8)

• Figure 1: Generated results based on instruction "Beginning with a beach scene, the camera gradually draws in closer as waves lap against the reef. Then the camera slowly pans right and a large area of sea is revealed". In Figure \ref{['fig:camera-static']}, the video footage is almost static, while in Figure \ref{['fig:inconsistency']}, the scene transitions show abrupt changes, and the scenes are mixed. Figure \ref{['fig:example']} shows the results of our proposed model.
• Figure 2: Framework for our proposed Modular-Cam, which contains four modules, i.e., Base Video Generator, CamOperator, AdaControlNet, and LLM-Director. First, the LLM is utilized to parse the user instruction, decomposing it into multiple scenes with descriptions. Then for each scene, a video generator is built, which has been integrated with CamOperator and AdaControlNet. LLM will identify the camera-view transformation in each scene and select from the MotionPool to plug in the appropriate CamOperator Module, which will enable the output video to follow the specific motion pattern, i.e., ZoomIn. A condition image, that is the ending frame of the last scene, is inputted into the AdaControlNet, which will guide the generation of the current scene. Finally, the video clips for each scene are concatenated orderly to form the final multi-scene dynamic camera-view video.
• Figure 3: Qualitative comparison between Modular-Cam and other baselines. We select several intermediate frames of the whole video for the convenience of presentation.
• Figure 4: Ablation study on adjusting the color tone, where Original represents the generated results of Modular-Cam, and RB and Ad stands for Randomized blending and Adaptive pixel normalization, respectively. We remove the two techniques and display the first frame of each generated video compared with the condition image, where areas with color tone shifting are marked with red frames.
• Figure 5: Ablation study on LLM decomposing user instruction. In Modular-Cam, the instruction is first parsed and decomposed by LLM then be fed to the video generator, while in Figure \ref{['fig:analysis-llm-no']}, we display the generated result of video generator directly utilizing the multi-scene involved instruction.