VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos

TL;DR

This work addresses the challenge of enabling diffusion-based video models to reproduce unconventional camera motions without relying on scarce real training data. It introduces ViVidCam, a dual-adaptation framework that learns appearance from synthetic, low-poly Unity scenes via an appearance LoRA and separately learns camera motion with a motion module, guided by an optical-flow loss and style-aligned prompts to prevent virtual appearance leakage. By rendering two synthetic datasets—$\mathcal{X}_a$ (static) and $\mathcal{X}_c$ (motion)—ViVidCam achieves precise, diverse camera motions across simple to complex categories, while maintaining realistic outputs comparable to baselines trained on real footage. The method reduces data collection burden and offers a scalable path for artistic and expressive video creation with controllable camera trajectories.

Abstract

Although recent text-to-video generative models are getting more capable of following external camera controls, imposed by either text descriptions or camera trajectories, they still struggle to generalize to unconventional camera motions, which is crucial in creating truly original and artistic videos. The challenge lies in the difficulty of finding sufficient training videos with the intended uncommon camera motions. To address this challenge, we propose VividCam, a training paradigm that enables diffusion models to learn complex camera motions from synthetic videos, releasing the reliance on collecting realistic training videos. VividCam incorporates multiple disentanglement strategies that isolates camera motion learning from synthetic appearance artifacts, ensuring more robust motion representation and mitigating domain shift. We demonstrate that our design synthesizes a wide range of precisely controlled and complex camera motions using surprisingly simple synthetic data. Notably, this synthetic data often consists of basic geometries within a low-poly 3D scene and can be efficiently rendered by engines like Unity. Our video results can be found in https://wuqiuche.github.io/VividCamDemoPage/ .

Figures (9)

• Figure 1: ViVidCam learns diverse unconventional camera motions from synthetic videos. The training data (1st row) are simple low-poly 3D scenes rendered in Unity in about 5 seconds per video. In contrast, the generated results show high visual quality with meaning-driven motions that convey intention (2nd–3rd rows) and more dramatic, unusual motions for artistic effect (4th–5th rows).
• Figure 2: State-of-the-art method bahmani2025ac3d fails to generate unconventional camera motions. More examples are in Appendix \ref{['appendix:qualitative']}.
• Figure 3: ViVidCam. We first render initial scene using publicly available assets. Then, we render videos with and without camera motion. These videos are used to train the camera and appearance modules, generating videos with desired camera motions. Details of training are in Sec. \ref{['method:lora']}.
• Figure 4: Qualitative results of diverse camera motions. From top to down: ❶ Push forward; ❷ Push forward, then truck right; ❸ Orbit shot; ❹ Pan around, then focus on one object; ❺ Switch focus between objects; ❻ Camera shaking; ❼ Camera rotating. The three panels correspond to simple, composed, and complex camera motions. Note that some complex camera motions are difficult to demonstrate through images; please refer to the videos on our webpage https://wuqiuche.github.io/VividCamDemoPage/ for better visual results.
• Figure 5: Visual examples illustrating the effects of appearance LoRA and style-aligned prompts.