Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training

TL;DR

This paper tackles the problem of enabling camera trajectory control in pre-trained video diffusion models without additional training. It introduces Latent-Reframe, a sampling-stage approach that reframes per-frame latent codes through time-aware 3D point clouds and repairs occluded regions via latent code inpainting and harmonization, preserving the model’s original distribution. The method delivers competitive or superior camera-control precision and video quality compared to training-based baselines, as evidenced by FID/FVD metrics and pose errors. By leveraging 3D information during inference, Latent-Reframe broadens controllable video generation while maintaining efficiency and fidelity to the pre-trained model.

Abstract

Precise camera pose control is crucial for video generation with diffusion models. Existing methods require fine-tuning with additional datasets containing paired videos and camera pose annotations, which are both data-intensive and computationally costly, and can disrupt the pre-trained model distribution. We introduce Latent-Reframe, which enables camera control in a pre-trained video diffusion model without fine-tuning. Unlike existing methods, Latent-Reframe operates during the sampling stage, maintaining efficiency while preserving the original model distribution. Our approach reframes the latent code of video frames to align with the input camera trajectory through time-aware point clouds. Latent code inpainting and harmonization then refine the model latent space, ensuring high-quality video generation. Experimental results demonstrate that Latent-Reframe achieves comparable or superior camera control precision and video quality to training-based methods, without the need for fine-tuning on additional datasets.

Figures (11)

• Figure 1: The proposed Latent-Reframe enables a text-to-video diffusion model to generate high-quality videos that accurately follow both the text prompt and a specified camera trajectory, all without requiring additional training.
• Figure 2: Overview of the proposed Latent-Reframe. In the middle of the denoising process of a pre-trained video diffusion model, we first extract time-aware 3D point cloud via a point cloud estimation model, which takes $x_0$ estimated by the halfway denoised latent code as the input. Next we reframe $x_0$ according to the target camera pose and the per-frame point cloud. Then we use the proposed latent space rehabilitation approach to inpaint the blank region due to occlusion and harmonize the latent code. $x_0^\prime$ is the reframed video after finishing the remaining denoising steps, which follows the target camera pose trajectory.
• Figure 3: Visual comparison with state-of-the-art methods. The proposed Latent-Reframe can generate videos following the given camera trajectory without training. The video quality and the camera pose accuracy are comparable with the compared training-based methods. AnimateDiff is the pre-trained text-to-video diffusion model used by all the compared method. Only Latent-Reframe can keep the learned video distribution of AnimateDiff.
• Figure 4: Comparison between the time-aware and time-static point clouds. Time-aware point cloud can capture more temporal dynamics of the video, For instance, the motion of the human face (row 1 and 2) and wave (row 3 and 4) are more prominent using time-aware point cloud, both are marked with red bounding boxes.
• Figure 5: Comparison between diffusion steps to apply Latent-Reframe. Using diffusion step $8$ out of $25$ allows for the reconstruction of high-precision point clouds while leaving enough room for latent space inpainting and harmonization.