Lightweight Multiplane Images Network for Real-Time Stereoscopic Conversion from Planar Video

TL;DR

This work tackles real-time planar-to-stereo conversion by learning a lightweight MPI representation without explicit depth maps. It introduces LMPIN, which combines a detail branch, a depth semantic branch, and a low-cost MPI renderer, aided by a training-time depth-aware auxiliary branch supervised by a pretrained depth model. It demonstrates competitive subjective quality to state-of-the-art MPI methods and contemporary TMPI while using far fewer parameters and achieving real-time performance at 2K resolutions, with substantial acceleration over TMPI in high-load scenarios. The approach reduces reliance on explicit depth inputs and improves robustness to occlusion, offering a practical solution for glasses-free displays and VR.

Abstract

With the rapid development of stereoscopic display technologies, especially glasses-free 3D screens, and virtual reality devices, stereoscopic conversion has become an important task to address the lack of high-quality stereoscopic image and video resources. Current stereoscopic conversion algorithms typically struggle to balance reconstruction performance and inference efficiency. This paper proposes a planar video real-time stereoscopic conversion network based on multi-plane images (MPI), which consists of a detail branch for generating MPI and a depth-semantic branch for perceiving depth information. Unlike models that depend on explicit depth map inputs, the proposed method employs a lightweight depth-semantic branch to extract depth-aware features implicitly. To optimize the lightweight branch, a heavy training but light inference strategy is adopted, which involves designing a coarse-to-fine auxiliary branch that is only used during the training stage. In addition, the proposed method simplifies the MPI rendering process for stereoscopic conversion scenarios to further accelerate the inference. Experimental results demonstrate that the proposed method can achieve comparable performance to some state-of-the-art (SOTA) models and support real-time inference at 2K resolution. Compared to the SOTA TMPI algorithm, the proposed method obtains similar subjective quality while achieving over $40\times$ inference acceleration.

Figures (7)

• Figure 1: The proposed lightweight multiplane images stereoscopic conversion network, (a) overall framework, (b) input image, (c) coarse depth map predicted by depth head, (d) visualization example of detail features, (e) fine-grained depth map, (f) 3D video conversion results of ADAMPI and the proposed method.
• Figure 2: Architecture of the proposed lightweight multiplane images stereoscopic conversion network.
• Figure 3: The structure of different blocks. (a) depth head, (b) mask head, (c) depth-guided enhanced feature fusion (DEFF) block, (d) basic residual blocks (RB)
• Figure 4: The planar-to-stereo conversion results of different methods on 3D movie test set.
• Figure 5: Comparative results of depth map generation using various methods.