ImmersePro: End-to-End Stereo Video Synthesis Via Implicit Disparity Learning

TL;DR

ImmersePro addresses the challenge of generating stereo video from monocular video by combining implicit disparity guidance with a layered disparity representation within a dual-branch, spatial-temporal Transformer framework. It jointly extracts disparity-focused and contextual features, propagates temporal information through multi-scale attention, and warps to produce smooth, semantically coherent right-view sequences without explicit depth maps. The work introduces Youtube-SBS, a public benchmark with over 7 million stereo pairs across 423 SBS videos, enabling robust training and evaluation. Empirically, ImmersePro achieves notable improvements over state-of-the-art stereo-from-mono methods in L1, SSIM, and PSNR, while maintaining temporal stability, though it relies on an average disparity estimate and faces limitations in generating very strong stereo effects; these insights point to future directions such as Nerf-based inpainting and larger, more diverse datasets.

Abstract

We introduce \textit{ImmersePro}, an innovative framework specifically designed to transform single-view videos into stereo videos. This framework utilizes a novel dual-branch architecture comprising a disparity branch and a context branch on video data by leveraging spatial-temporal attention mechanisms. \textit{ImmersePro} employs implicit disparity guidance, enabling the generation of stereo pairs from video sequences without the need for explicit disparity maps, thus reducing potential errors associated with disparity estimation models. In addition to the technical advancements, we introduce the YouTube-SBS dataset, a comprehensive collection of 423 stereo videos sourced from YouTube. This dataset is unprecedented in its scale, featuring over 7 million stereo pairs, and is designed to facilitate training and benchmarking of stereo video generation models. Our experiments demonstrate the effectiveness of \textit{ImmersePro} in producing high-quality stereo videos, offering significant improvements over existing methods. Compared to the best competitor stereo-from-mono we quantitatively improve the results by 11.76\% (L1), 6.39\% (SSIM), and 5.10\% (PSNR).

Figures (8)

• Figure 1: ImmersePro is a video method to convert a single-view video to a stereo video by predicting plausible right-view images for each input frame. Compared to previous work processing images frame by frame (3D Photo or Stereo from Mono), our method has the best visual quality.
• Figure 2: Illustration of ImmersePro framework. Our network contains six parts: (1) dual-branch feature extractors for extracting disparity features and context features (\ref{['sec:dual_branch']}), (2) multi-scale spatial-temporal self-attention to refine disparity features (\ref{['sec:attention']}), (3) implicit disparity to generate stereo images without explicit disparities (\ref{['sec:implicit_disp']}), (4) spatial-temporal cross attention block to inject contextual information into the implicit disparity features (\ref{['sec:attention']}), (5) layered disparity to obtain the estimated right view video sequences (\ref{['sec:layered_disp']}), and (6) context fusion to enrich the estimated right view video sequences with detailed semantic information (\ref{['sec:context_fusion']}).
• Figure 3: Visual demonstration of the implicit disparity guidance. We can observe that (1) the implicit disparity module tries to resolve the disparity from the given image, and (2) our method can significantly rectify the error introduced by the implicit disparity estimations. Our method offers a significant improvement regarding clarity with less irregular texture deformation on the image. The implicit disparity map contains multiple channels and we apply $argmax$ to obtain the visual output.
• Figure 4: Visual demonstration of our layered disparity representation. We show the $1^{st}$ and $3^{rd}$ disparity layers in \ref{['fig:layered-4', 'fig:layered-5']}. We denote darker colors as moving to the right and lighter colors as moving to the left. We use 7 layers in total while we found the $1^{st}$ and $3^{rd}$ layers contribute to the right-view generation most. We observe that the $1^{st}$ layer aims to warp the majority of the pixels to the right to their correct right-view location while the $3^{rd}$ layer moves pixels to the left to fill the resulting holes, e.g. near the left border.
• Figure 5: Visual demonstration of the disparity analysis results. Our network predicts reasonable stereo effects but may be stronger or weaker if compared to the ground truth. The L2R disparity computes the left-to-right disparity using RAFT-Stereo lipson2021raft.