Eye2Eye: A Simple Approach for Monocular-to-Stereo Video Synthesis

TL;DR

Eye2Eye introduces a direct mono-to-stereo video synthesis framework that leverages a pre-trained cascaded video diffusion model to generate a left-eye view from a right-eye monocular video, avoiding explicit depth estimation and warping. The method uses a two-stage pipeline (base low-resolution generation and high-resolution refinement) trained on real stereo data (Stereo4D) and demonstrates superior performance in scenes with reflections and complex light transport, as shown by a user study and the iSQoE metric. By learning from real stereo video priors and treating stereo content as a single coherent synthesis problem, Eye2Eye achieves robust 3D effects across diverse content without multi-layer depth decomposition. The work highlights the growing capability of large video models to internalize geometric and photometric priors relevant to stereo rendering, offering practical potential for VR and immersive media production.

Abstract

The rising popularity of immersive visual experiences has increased interest in stereoscopic 3D video generation. Despite significant advances in video synthesis, creating 3D videos remains challenging due to the relative scarcity of 3D video data. We propose a simple approach for transforming a text-to-video generator into a video-to-stereo generator. Given an input video, our framework automatically produces the video frames from a shifted viewpoint, enabling a compelling 3D effect. Prior and concurrent approaches for this task typically operate in multiple phases, first estimating video disparity or depth, then warping the video accordingly to produce a second view, and finally inpainting the disoccluded regions. This approach inherently fails when the scene involves specular surfaces or transparent objects. In such cases, single-layer disparity estimation is insufficient, resulting in artifacts and incorrect pixel shifts during warping. Our work bypasses these restrictions by directly synthesizing the new viewpoint, avoiding any intermediate steps. This is achieved by leveraging a pre-trained video model's priors on geometry, object materials, optics, and semantics, without relying on external geometry models or manually disentangling geometry from the synthesis process. We demonstrate the advantages of our approach in complex, real-world scenarios featuring diverse object materials and compositions. See videos on https://video-eye2eye.github.io

Figures (7)

• Figure 1: 3D anaglyph visualization of stereo videos produced by our method. Our framework, Eye2Eye, takes as input a monocular video representing a right-eye view (top), and produces a left-eye video (visualized in the anaglyph on the bottom), enabling stereoscopic viewing using 3D glasses or a VR headset. Our method directly produces the new viewpoint, avoiding steps like explicit depth estimation and warping, and thus can plausibly handle challenging scenes with specular or transparent surfaces, such as the wine glass in the left example or the shiny floor in the right example, where assumptions of a single well-defined depth per pixel do not hold.
• Figure 2: Limitations of warp-and-inpaint approach for mono-to-stereo video synthesis. Given an input video frame (left), we use a state-of-the-art disparity estimation model hu2024-DepthCrafter to compute its disparity (middle), and use it to warp the original frame to a new view (right). Since the predicted disparity map captures only the surface of the table, without considering the reflection of other objects off of it, the warped frame depicts incorrect reflection (skewed diagonally instead of reflecting vertically). When viewed in VR, the reflection on the table appears "flat", as if it is a part of the table. This demonstrates the fundamental limitation of the common warp-and-inpaint approach for stereo view synthesis.
• Figure 3: Data processing pipeline. We curate stereo VR180 footage captured with high-resolution cameras and stored in a equirectangular format. Following Stereo4D jin2024stereo4d, we rectify the stereo videos and map the equirectangular format to perspective videos. We filter out videos with large disparity using RAFT teed2020raft and caption the remaining videos with BLIP2 li2023blip.
• Figure 4: Eye2Eye mono-to-stereo pipeline. We leverage the pre-trained Lumiere cascaded text-to-video model, as well as a curated dataset of rectified stereo pairs, to perform mono-to-stereo synthesis. We finetune two different copies of a base (low-resolution) pre-trained Lumiere model, in two different contexts. For the first base model, we add additional input channels to condition the model on an input right eye, and train the base Eye2Eye generator on downsampled, low-resolution 128$\times$128 stereo pairs (top left). We call the resulting trained model the base Eye2Eye generator model. We train the second model to be a refinement model with the same conditioning mechanism, but instead trained on 128$\times$128 crops from full, high-resolution images (bottom left). We call the resulting model the Eye2Eye refiner model. The base Eye2Eye model models correct pixels disparity at a low resolution, and the Eye2Eye refiner has better quality in inpainted areas or areas with large disparities. At inference time, our sampling process (right) combines both models' strengths by first generating a low-resolution output from the base Eye2Eye model to establish appropriate stereo disparity for a compelling 3D effect, then noising and denoising it with the Eye2Eye refiner to achieve high visual quality.
• Figure 5: Resolving training and inference gap. We ablate the use of the two models in our pipeline, illustrating the training-inference gap of each of them, and visualize the resulting anaglyph and depth estimation (estimated using teed2020raft) of their outputs. When sampling from the Eye2Eye-refiner model (trained on crops without any downsampling), far away content is still shifted by a large amount (column 1). When sampling from the base Eye2Eye generator at a higher resolution than its training resolution, the scale of the disparity and novel content in the frame reduces, weakening the 3D effect compared to sampling at the training resolution (columns 2 and 3, in column 2 the outputs were upsampled). By upsampling the outputs of the base stereo model and noising and denoising it with the Eye2Eye-refiner model, we maintain both a good depth perception from the base model and the stereo refiner's ability to generate high quality frames (column 4).