CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation

TL;DR

Presents CineMaster, a 3D-aware, controllable text-to-video framework with a two-stage pipeline: an interactive 3D workflow to define object and camera motion, and a depth-guided diffusion model conditioned on 3D layouts and trajectories. It introduces Semantic Layout ControlNet and Camera Adapter to fuse depth maps and camera poses into video synthesis, supported by an automated data-labeling pipeline to obtain 3D annotations at scale. Experiments show state-of-the-art controllability and quality, validated via quantitative metrics and ablations. This work enables filmmaker-like control in text-to-video generation and broadens open-world 3D-aware video synthesis.

Abstract

In this work, we present CineMaster, a novel framework for 3D-aware and controllable text-to-video generation. Our goal is to empower users with comparable controllability as professional film directors: precise placement of objects within the scene, flexible manipulation of both objects and camera in 3D space, and intuitive layout control over the rendered frames. To achieve this, CineMaster operates in two stages. In the first stage, we design an interactive workflow that allows users to intuitively construct 3D-aware conditional signals by positioning object bounding boxes and defining camera movements within the 3D space. In the second stage, these control signals--comprising rendered depth maps, camera trajectories and object class labels--serve as the guidance for a text-to-video diffusion model, ensuring to generate the user-intended video content. Furthermore, to overcome the scarcity of in-the-wild datasets with 3D object motion and camera pose annotations, we carefully establish an automated data annotation pipeline that extracts 3D bounding boxes and camera trajectories from large-scale video data. Extensive qualitative and quantitative experiments demonstrate that CineMaster significantly outperforms existing methods and implements prominent 3D-aware text-to-video generation. Project page: https://cinemaster-dev.github.io/.

Figures (5)

• Figure 1: CineMaster targets at granting users 3D-aware and intuitive control over the text-to-video generation process. We first design a 3D-native workflow that enables users to manipulate objects and camera in the 3D space. Then the rendered depth maps and camera trajectories serve as strong guidance to synthesize the desired video content. Left column shows the objects and camera setup using the proposed workflow. Right columns indicate synthesized frames with rendered depth maps on the bottom left.
• Figure 2: Overview of CineMaster. CineMaster consists of two stages. First, we present an interactive workflow that allows users to intuitively manipulate the objects and camera in a 3D-native manner. Then the control signals are rendered from the 3D engine and fed into a text-to-video diffusion model, guiding the generation of user-intended video content.
• Figure 3: Overview of the network architecture. We design a Semantic Layout ControlNet which consists of a semantic injector and a DiT-based ControlNet. Semantic injector fuses the 3D spatial layout and class label conditions. The DiT-based ControlNet further represents the fused features and adds to the hidden states of the base model. Meanwhile, we inject the camera trajectories by the camera adapter to achieve joint control over object motion and camera motion.
• Figure 4: Dataset Labeling Pipeline. We propose a data labeling pipeline to extract 3D bounding boxes, class labels and camera poses from videos. Our pipeline consists of four steps: 1) Instance Segmentation: Obtain instance segmentation results from the foreground in videos. 2) Depth Estimation: Produce metric depth maps using DepthAnything V2. 3) 3D Point Cloud and Box Calculation: Identify the frame with the largest mask for each entity and compute the 3D point cloud of each entity through inverse projection. Then, use the minimum volume method to calculate the 3D bounding box for each entity. 4) Entity Tracking and 3D Box Adjustment: Access the point tracking results of each entity and calculate the 3D bounding boxes for each frame. Finally, project the entire 3D scene into depth maps.
• Figure 5: We present three different feature comparisons: moving object $\&$ static camera, static object $\&$ moving camera and moving object $\&$ moving camera. We transform our 3D box condition to object trajectories for MotionCtrl MotionCtrl and 2D bounding box sequences for Direct-A-Video yang2024direct to align the input conditions. In comparison, CineMaster could better control object motion and camera motion separately or jointly to generate diverse user-intended scenes.