SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input

TL;DR

SpatialDreamer addresses monocular stereo video synthesis by combining a Depth based Video Generation (DVG) module with a RefinerNet in a self-supervised diffusion framework. A Consistency Control Module, featuring Temporal Interaction Learning (TIL) and stereo deviation strength, enforces geometric and temporal coherence across frames. The method uses forward-backward rendering with optical-flow–driven occlusion refinement to generate paired videos for training and employs a two-stage training strategy to capture both frame-wise and temporal context. Empirical results on RealEstate10K and a self-collected monocular dataset show state-of-the-art performance in both stereo image and stereo video synthesis, with strong temporal stability and competitive computational characteristics for diffusion-based video generation. The approach demonstrates strong potential for VR and immersive media applications where high-quality, consistent stereo content is required.

Abstract

Stereo video synthesis from a monocular input is a demanding task in the fields of spatial computing and virtual reality. The main challenges of this task lie on the insufficiency of high-quality paired stereo videos for training and the difficulty of maintaining the spatio-temporal consistency between frames. Existing methods primarily address these issues by directly applying novel view synthesis (NVS) techniques to video, while facing limitations such as the inability to effectively represent dynamic scenes and the requirement for large amounts of training data. In this paper, we introduce a novel self-supervised stereo video synthesis paradigm via a video diffusion model, termed SpatialDreamer, which meets the challenges head-on. Firstly, to address the stereo video data insufficiency, we propose a Depth based Video Generation module DVG, which employs a forward-backward rendering mechanism to generate paired videos with geometric and temporal priors. Leveraging data generated by DVG, we propose RefinerNet along with a self-supervised synthetic framework designed to facilitate efficient and dedicated training. More importantly, we devise a consistency control module, which consists of a metric of stereo deviation strength and a Temporal Interaction Learning module TIL for geometric and temporal consistency ensurance respectively. We evaluated the proposed method against various benchmark methods, with the results showcasing its superior performance.

Figures (13)

• Figure 1: Visualization of the temporal consistency in stereo video generated by different methods. We extract the yellow line of each frame and stack them together. A good result should show a natural transition in the $t$ dimension.
• Figure 2: Overview of the proposed method, given left view video, the target view video is rendered, encoded, and concatenated with multi-frame noise, followed by the denoising U-Net architecture (i.e., SVD). The reference view images are fed into RefinerNet, through which the spatial features are extracted. (Notably, the left and right views are the target and reference view for both training and inference). The temporal interaction learning module integrates the latent features from long-temporal frames, and the deviation strength is projected as positional embedding and added to the time step embedding. Finally, the variational autoencoder decoder decodes the result into a video clip.
• Figure 3: DVG. The temporal motion can be utilized to refine the occluded regions in the current frame, thereby providing smoother images and more consistent occlusion in the temporal sequence.
• Figure 4: Stereo deviation strength guidance examples. Augmenting the deviation strength enhances the 3D photography effect, but excessive deviation strength may lead to image distortions or even lower the quality of stereoscopic image.
• Figure 5: Qualitative results on the RealEstate10K dataset. The proposed method generates better-quality information and maintains the edge and textural details of the reference image better than the other methods. More results are available in supplementary material.