ChronosObserver: Taming 4D World with Hyperspace Diffusion Sampling

TL;DR

ChronosObserver tackles the problem of generating 3D-consistent, time-synchronized multi-view videos from a single monocular input without training diffusion models.It introduces World State Hyperspace to encode incremental spatiotemporal constraints and Hyperspace Guided Sampling to steer diffusion trajectories across views, enforcing a unified 4D scene.The method is training-free and leverages a pre-trained camera-controlled diffusion model, augmented by an incremental state representation derived from depth and pose information.Through experiments on a 30-video dataset, ChronosObserver demonstrates notable improvements in 3D consistency and video quality over state-of-the-art baselines, including robustness to missing data and extrapolated viewpoints.

Abstract

Although prevailing camera-controlled video generation models can produce cinematic results, lifting them directly to the generation of 3D-consistent and high-fidelity time-synchronized multi-view videos remains challenging, which is a pivotal capability for taming 4D worlds. Some works resort to data augmentation or test-time optimization, but these strategies are constrained by limited model generalization and scalability issues. To this end, we propose ChronosObserver, a training-free method including World State Hyperspace to represent the spatiotemporal constraints of a 4D world scene, and Hyperspace Guided Sampling to synchronize the diffusion sampling trajectories of multiple views using the hyperspace. Experimental results demonstrate that our method achieves high-fidelity and 3D-consistent time-synchronized multi-view videos generation without training or fine-tuning for diffusion models.

Figures (15)

• Figure 1: ChronosObserver results of temporal-synchronized multi-view videos from one single monocular video. The project page is https://icvteam.github.io/ChronosObserver.html.
• Figure 2: Motivation. Directly lifting a camera-controlled video generation method trajectorycrafter to generate multi-view videos leads to 3D inconsistencies across different viewpoints at the same timestamp.
• Figure 3: ChronosObserver Pipeline starts from the monocular input video and generates time-synchronized multi-view videos. ChronosObserver incrementally constructs the World State Hyperspace and utilizes it for the Hyperspace Guided Sampling.
• Figure 4: Intuitive Illustration of ChronosObserver compared to camera-controlled video generation methods.
• Figure 5: Qualitative comparisons with other methods (TrajectoryCrafter trajectorycrafter, EX-4D ex4d, Reangle-A-Video rav, ViewCrafter viewcrafter).