studentSplat: Your Student Model Learns Single-view 3D Gaussian Splatting

TL;DR

Two techniques are introduced: 1) a teacher-student architecture where a multi-view teacher model provides geometric supervision to the single-view student during training, addressing scale ambiguity and encourage geometric validity; and 2) an extrapolation network that completes missing scene context, enabling high-quality extrapolation.

Abstract

Recent advance in feed-forward 3D Gaussian splatting has enable remarkable multi-view 3D scene reconstruction or single-view 3D object reconstruction but single-view 3D scene reconstruction remain under-explored due to inherited ambiguity in single-view. We present \textbf{studentSplat}, a single-view 3D Gaussian splatting method for scene reconstruction. To overcome the scale ambiguity and extrapolation problems inherent in novel-view supervision from a single input, we introduce two techniques: 1) a teacher-student architecture where a multi-view teacher model provides geometric supervision to the single-view student during training, addressing scale ambiguity and encourage geometric validity; and 2) an extrapolation network that completes missing scene context, enabling high-quality extrapolation. Extensive experiments show studentSplat achieves state-of-the-art single-view novel-view reconstruction quality and comparable performance to multi-view methods at the scene level. Furthermore, studentSplat demonstrates competitive performance as a self-supervised single-view depth estimation method, highlighting its potential for general single-view 3D understanding tasks.

Figures (14)

• Figure 1: What can you do with studentSplat? All the results are generated using our studentSplat with teacher refine (detailed in the Appendix \ref{['sec:teacher_refine']}) with only one input image. The input to our studentSplat is highlighted in green. studentSplat directly takes the generated image from Stable Diffusion rombach2022high in text-to-3D application.
• Figure 2: The training pipeline of studentSplat. The multi-view teacher network is used during training to produce 3D Gaussians centers (up-to an unknown scale) for geometric supervision. The input to student model is highlighted in green. The rendered student output is processed through the Extrapolator before performing novel-view supervision.
• Figure 3: The qualitative comparison between representative methods in the extrapolation setting. Top two rows are from RE10K and the bottom two rows are from ACID The multi-view method use both of the context views whereas the single-view method only use the context view highlighted in green. Additional examples are in the Appendix \ref{['sec:add_results']}.
• Figure 4: The qualitative comparison between representative methods in the single-view cross-dataset generalization setting. The context view is highlighted in green.
• Figure 5: The qualitative ablation results. The input view is highlighted in green. The ground truth target view is below the input view. We zoomed in some areas for better comparison.