CamViG: Camera Aware Image-to-Video Generation with Multimodal Transformers

TL;DR

CamViG addresses the challenge of controlling 3D camera motion during image-to-video generation by conditioning a token-based video transformer on a discretized 3D camera path in addition to a single input frame $I$ and generating an $n$-frame sequence with $F_1=I$. The method introduces a camera-path modality encoded as audio-like tokens and fuses it with the video tokens, built on a pre-trained video-poet style backbone; training data are synthesized via Neural Radiance Fields to provide grounded camera trajectories. Key findings show that the model can follow explicit camera instructions and produce plausible parallax while maintaining dynamic scene content, with in-/out-painting arising naturally from the generative process. A mixed data strategy, combining roughly 70% NeRF-scene data with 30% broad video data, yields the best trade-off between accurate camera following and preserving scene motion, suggesting practical relevance for camera-aware video synthesis from a single image. Overall, CamViG advances toward controllable, generalizable camera-aware video generation without explicit geometry, enabling applications in film, AR/VR, and content creation where precise camera movements are desired from minimal inputs.

Abstract

We extend multimodal transformers to include 3D camera motion as a conditioning signal for the task of video generation. Generative video models are becoming increasingly powerful, thus focusing research efforts on methods of controlling the output of such models. We propose to add virtual 3D camera controls to generative video methods by conditioning generated video on an encoding of three-dimensional camera movement over the course of the generated video. Results demonstrate that we are (1) able to successfully control the camera during video generation, starting from a single frame and a camera signal, and (2) we demonstrate the accuracy of the generated 3D camera paths using traditional computer vision methods.

Figures (5)

• Figure 0: Our method generates videos conditioned on a single image and a 3D camera path. By default, video generation models may move the "implicit camera" viewing the scene in unpredictable ways as in (a). Our method gives explicit control over 3D camera movement, while inducing parallax (despite no explicit 3D or depth representation of the scene) and retaining the inherent inpainting and outpainting abilities of a pre-trained video generation model. In our output (b) above, notice the changing relative positions of flowers against the horizon, and the outpainting at the bottom of the frame as the camera translates downward in 3D.
• Figure 1: Model Architecture. We train a video transformer that takes as input a single frame's worth of visual tokens, plus camera path tokens for the entire length of video. The output is a set of video tokens that are detokenized into a 17-frame video conditioned on the input camera path. The tokenizer and detokenizer are frozen, while the video transformer is fully trained to implement our method.
• Figure 2: NeRF-Based Training Data. Video sequences with corresponding ground truth camera paths are used to train our model. We render 10,000 short clips each from large NeRF scenes -- by rendering clips of cameras moving in all directions, we remove correlation between scene content and camera motion during training.
• Figure 3: Optical flow MSE (Mean Squared Error) between generated video outputs and ground truth video from the holdout validation set. We show results over the course of training, and for models that use a mixture of 70% NeRF scenes and 30% unconstrained video (blue), a mixture of 80% and 20% (red), and a model which uses only data from NeRF scenes (yellow). During training, decreasing the difference in optical flow between the generated and ground-truth videos indicates better following of camera movement instructions. We see that as training progresses, overall optical flow MSE decreases. Eventually the model over-fits to the NeRF dataset, resulting in lower quality videos on the unseen validation set which causes a decrease in performance.
• Figure 4: A series of four generated videos alongside the ground truth on the holdout validation set. The model is instructed to move the camera left in the top two scenes, and down in the bottom two scenes. We see the model correctly follows the speed and direction instructed while generating convincing outputs. In the first scene (top left), the model correctly induces parallax, flattening the profile of the bookcase and occluding the bed and dresser. We also see correct object generation in the case of finishing the chair legs (bottom right) and the door (top right).