SparSplat: Fast Multi-View Reconstruction with Generalizable 2D Gaussian Splatting

TL;DR

SparSplat tackles the problem of fast, generalizable 3D reconstruction and novel-view synthesis from sparse views by predicting 2D Gaussian splats in a feed-forward manner. It uses a two-branch pipeline that fuses MVS-based depth estimation with pixel-aligned splat attribute regression, and leverages deep features from 2D and 3D foundation models (DINOv2 and MASt3R) to improve generalization. The approach achieves state-of-the-art reconstruction on the DTU sparse-view benchmark and strong generalization on BlendedMVS and Tanks and Temples, while also delivering state-of-the-art NVS in the sparse DTU setting, and runs with inference times around 0.88s, orders of magnitude faster than prior implicit methods. The work demonstrates practical impact for real-time 3D reconstruction and view synthesis in robotics, AR/VR, and related applications, by combining efficiency, accuracy, and cross-scene generalizability.

Abstract

Recovering 3D information from scenes via multi-view stereo reconstruction (MVS) and novel view synthesis (NVS) is inherently challenging, particularly in scenarios involving sparse-view setups. The advent of 3D Gaussian Splatting (3DGS) enabled real-time, photorealistic NVS. Following this, 2D Gaussian Splatting (2DGS) leveraged perspective accurate 2D Gaussian primitive rasterization to achieve accurate geometry representation during rendering, improving 3D scene reconstruction while maintaining real-time performance. Recent approaches have tackled the problem of sparse real-time NVS using 3DGS within a generalizable, MVS-based learning framework to regress 3D Gaussian parameters. Our work extends this line of research by addressing the challenge of generalizable sparse 3D reconstruction and NVS jointly, and manages to perform successfully at both tasks. We propose an MVS-based learning pipeline that regresses 2DGS surface element parameters in a feed-forward fashion to perform 3D shape reconstruction and NVS from sparse-view images. We further show that our generalizable pipeline can benefit from preexisting foundational multi-view deep visual features. The resulting model attains the state-of-the-art results on the DTU sparse 3D reconstruction benchmark in terms of Chamfer distance to ground-truth, as-well as state-of-the-art NVS. It also demonstrates strong generalization on the BlendedMVS and Tanks and Temples datasets. We note that our model outperforms the prior state-of-the-art in feed-forward sparse view reconstruction based on volume rendering of implicit representations, while offering an almost 2 orders of magnitude higher inference speed.

Figures (6)

• Figure 1: We present the first generalizable feed-forward 2DGS prediction model from multi-view images. It achieves state-of-the-art performance in the sparse DTU aanaes2016large 3D reconstruction benchmark long2022sparseneus, with faster inference by several orders of magnitude compared to the competition based on volume rendering of implicit representations. Multi-view input deep features are homography-warped into the target view. A two-fold network performs Deep Multi-view Stereo and pixel aligned 2D surface element attribute regression. Perspective accurate Gaussian Splatting huang20242d of these surface elements enables real-time novel view synthesis and mesh extraction.
• Figure 2: Qualitative comparison of reconstructions from 3 input views in datatset DTU. (Inference time: Ours 0.8s, UfoReconren2023volrecon 66s).
• Figure 3: Qualitative comparison of reconstructions from 3 input views in datatset BMVS. Note that we reconstruct detailed surfaces with our method without any fine-tuning.
• Figure 4: Qualitative results of reconstructions from 3 input views in datatset Tanks and Temples using our method without any fine-tuning. Notice that these are very challenging outdoor cases.
• Figure 5: Novel-View synthesis qualitative evalutation on DTU aanaes2016large using $3$ source images. We outperform SOTA MVSGaussian liu2024fast and exhibit sharper view synthesis results in regions with low input view overlap