TranSplat: Generalizable 3D Gaussian Splatting from Sparse Multi-View Images with Transformers

TL;DR

TranSplat introduces a transformer-driven, generalizable sparse-view 3D Gaussian Splatting method that predicts per-pixel Gaussian parameters from multi-view RGB inputs. Key innovations include the Depth-aware Deformable Matching Transformer (DDMT) for robust depth distribution and a Depth Refine U-Net that leverages monocular depth priors to correct non-overlapping regions. The approach yields state-of-the-art results on RealEstate10K and ACID, with strong cross-dataset generalization to DTU, while maintaining competitive speed. Ablation studies verify the necessity of DDMT, depth priors, and camera-parameter encoding, and reveal trade-offs in depth-prior and model-size choices. Overall, TranSplat enables efficient, high-quality 3D reconstruction from sparse views with robust generalization across datasets.

Abstract

Compared with previous 3D reconstruction methods like Nerf, recent Generalizable 3D Gaussian Splatting (G-3DGS) methods demonstrate impressive efficiency even in the sparse-view setting. However, the promising reconstruction performance of existing G-3DGS methods relies heavily on accurate multi-view feature matching, which is quite challenging. Especially for the scenes that have many non-overlapping areas between various views and contain numerous similar regions, the matching performance of existing methods is poor and the reconstruction precision is limited. To address this problem, we develop a strategy that utilizes a predicted depth confidence map to guide accurate local feature matching. In addition, we propose to utilize the knowledge of existing monocular depth estimation models as prior to boost the depth estimation precision in non-overlapping areas between views. Combining the proposed strategies, we present a novel G-3DGS method named TranSplat, which obtains the best performance on both the RealEstate10K and ACID benchmarks while maintaining competitive speed and presenting strong cross-dataset generalization ability. Our code, and demos will be available at: https://xingyoujun.github.io/transplat.

Figures (10)

• Figure 1: Given two-view images, TranSplat achieves higher quality in both novel view synthesis and 3D Gaussian construction, particularly in challenging areas such as low-texture, repetitive patterns, and non-overlapping regions, compared to MVSplat chen2024mvsplat. Our novel view synthesis results contain fewer artifacts and maintain better geometric consistency.
• Figure 2: Framework of TranSplat. Our method takes multi-view images as input and first extracts image features and monocular depth priors. Next, the coarse-to-fine matching stage is used to obtain a geometry-consistent depth distribution for each view. Specifically, we compute multi-view feature similarities using our proposed Depth-Aware Deformable Matching Transformer module. The Depth Refine U-Net is then employed to further refine the depth prediction. Finally, we predict pixel-wise 3D Gaussian parameters to render novel views.
• Figure 3: The details of Deformable Sampling module. We sample cross-view points using deformable attention, which enhances the aggregation of local spatial information.
• Figure 4: The illustration of Depth-aware Matching Transformer. We utilize the depth confidence map to integrate into our Depth-aware Matching Transformer, enabling the network to prioritize regions with higher depth confidence and improving the overall depth prediction accuracy. (Here D.A. feature represents depth-aware feature.)
• Figure 5: The qualitative comparisons with SOTA methods. Our method outperforms other state-of-the-art methods across various scenes, with the first three rows from RealEstate10K and the last one from ACID. TranSplat excels in challenging regions thanks to the effectiveness of our transformer-based Depth-Aware Deformable Matching Transformer (DDMT).