SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time

TL;DR

SpaceTimePilot tackles the problem of generating controllable 4D renderings from monocular video by disentangling space (camera viewpoint) and time (scene dynamics). It introduces an animation time embedding and a temporal-warping training scheme to learn independent temporal control, plus a source-aware camera conditioning mechanism and the Cam×Time synthetic dataset for dense spatiotemporal supervision. The method supports retiming, new viewpoints, bullet-time, slow motion, and arbitrary space-time trajectories, including long-range generation via autoregressive segmentation. Experiments show superior disentanglement and control over baselines on real and synthetic data, with practical implications for video editing and 4D scene rendering.

Abstract

We present SpaceTimePilot, a video diffusion model that disentangles space and time for controllable generative rendering. Given a monocular video, SpaceTimePilot can independently alter the camera viewpoint and the motion sequence within the generative process, re-rendering the scene for continuous and arbitrary exploration across space and time. To achieve this, we introduce an effective animation time-embedding mechanism in the diffusion process, allowing explicit control of the output video's motion sequence with respect to that of the source video. As no datasets provide paired videos of the same dynamic scene with continuous temporal variations, we propose a simple yet effective temporal-warping training scheme that repurposes existing multi-view datasets to mimic temporal differences. This strategy effectively supervises the model to learn temporal control and achieve robust space-time disentanglement. To further enhance the precision of dual control, we introduce two additional components: an improved camera-conditioning mechanism that allows altering the camera from the first frame, and CamxTime, the first synthetic space-and-time full-coverage rendering dataset that provides fully free space-time video trajectories within a scene. Joint training on the temporal-warping scheme and the CamxTime dataset yields more precise temporal control. We evaluate SpaceTimePilot on both real-world and synthetic data, demonstrating clear space-time disentanglement and strong results compared to prior work. Project page: https://zheninghuang.github.io/Space-Time-Pilot/ Code: https://github.com/ZheningHuang/spacetimepilot

Figures (15)

• Figure 1: SpaceTimePilot enables unified control over both camera and time within a single diffusion model, producing continuous and coherent videos along arbitrary space–time trajectories. Given a source video (odd rows), our model synthesizes new videos (even rows) with retimed motion sequences, including slow motion, reverse motion, and bullet time, while precisely controlling camera movement according to a given camera trajectory.
• Figure 2: Space–time controllability across methods. Blue cells denote the input video/views, while arrows and dots indicate generated continuous videos or sparse frames. Camera-control V2V models Bai2025vanhoorick2024gcd modify only the camera trajectory while keeping time strictly monotonic. 4D multi-view models Wu2024liang2024diffusion4d synthesize discrete sparse views conditioned on space and time, but do not generate continuous video sequences. SpaceTimePilot enables free movement along both the camera and time axes with full control over direction and speed, supporting bullet-time, slow-motion, reverse playback, and mixed space–time trajectories.
• Figure 3: Temporal Wrapping for Spatiotemporal Disentanglement. (Top) For multi-view dynamic scene datasets Bai2025, a set of temporal warping operations (e.g. reverse playback, zigzag motion, slow motion, and freeze) are applied to the target video, with the source video kept as the standard forward reference, providing explicit supervision for temporal control . (Bottom) Compared with existing camera-control vanhoorick2024gcdBai2025 and joint-dataset training strategies Wu2024watson2025controllingspacetimediffusion, which rely on monotonic time progression and static-scene videos to demonstrate temporal differences, Temporal Wrapping provide much more diverse and explicit signals of temporal variation, leading to disentanglement of space and time.
• Figure 4: Cam$\times$Time dataset visualization. (Top) A space-time grid defined by a camera trajectory $\mathbf{c}=[c_1,...,c_F]$ and animation status $\mathbf{t}=[t_1,...,t_F]$. Cam$\times$Time renders images for all $(c,t)$ pairs, covering the full grid for learning disentangled spatial and temporal control. Any two sampled sequences of $F$ frames from the grid can form a source-target pair. (Bottom) One typical choice of source videos is taking the diagonal cells in green.
• Figure 5: Qualitative results of SpaceTimePilot. Our model enables fully disentangled control over camera motion and temporal dynamics. Each row shows a different combination of camera trajectory (left icons) and temporal warping (right icons). SpaceTimePilot produces coherent videos under diverse controls, including normal playback, reverse playback, bullet-time, slow-motion, replay motion, and complex camera paths (pan, tilt, zoom, and vertical motion).