NexusGS: Sparse View Synthesis with Epipolar Depth Priors in 3D Gaussian Splatting

TL;DR

NexusGS tackles sparse-view 3D Gaussian Splatting by embedding epipolar depth priors directly into the initial point cloud. Through Epipolar Depth Nexus, Flow-Resilient Depth Blending, and Flow-Filtered Depth Pruning, it computes dense, depth-consistent priors from optical flow and camera poses, enabling stable training without depth regularization. The approach delivers state-of-the-art depth accuracy and rendering quality across LLFF, Blender, DTU, and Mip-NeRF-360, and its dense priors transfer benefits to competing methods, highlighting broad applicability. The results demonstrate that leveraging epipolar geometry can significantly improve sparse-view 3D scene reconstruction and rendering fidelity in both object-centric and real-world scenes.

Abstract

Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) have noticeably advanced photo-realistic novel view synthesis using images from densely spaced camera viewpoints. However, these methods struggle in few-shot scenarios due to limited supervision. In this paper, we present NexusGS, a 3DGS-based approach that enhances novel view synthesis from sparse-view images by directly embedding depth information into point clouds, without relying on complex manual regularizations. Exploiting the inherent epipolar geometry of 3DGS, our method introduces a novel point cloud densification strategy that initializes 3DGS with a dense point cloud, reducing randomness in point placement while preventing over-smoothing and overfitting. Specifically, NexusGS comprises three key steps: Epipolar Depth Nexus, Flow-Resilient Depth Blending, and Flow-Filtered Depth Pruning. These steps leverage optical flow and camera poses to compute accurate depth maps, while mitigating the inaccuracies often associated with optical flow. By incorporating epipolar depth priors, NexusGS ensures reliable dense point cloud coverage and supports stable 3DGS training under sparse-view conditions. Experiments demonstrate that NexusGS significantly enhances depth accuracy and rendering quality, surpassing state-of-the-art methods by a considerable margin. Furthermore, we validate the superiority of our generated point clouds by substantially boosting the performance of competing methods. Project page: https://usmizuki.github.io/NexusGS/.

Figures (13)

• Figure 1: Given a few input images, our method first computes depth using optical flow and camera poses at the Epipolar Depth Nexus step. We then fuse depth values from different views, minimizing flow errors with flow-resilient depth blending. Before forming the final dense point cloud, outlier depths are removed at the flow-filter depth pruning step. In training, we do not need depth regularization, thanks to the embedded epipolar depth prior in the point cloud.
• Figure 2: We summarize potential depth blending error scenarios and compare the strategies of selecting the minimum depth versus the average depth. In (a), (1) illustrates the case where two different depth values (blue points) calculated from two views are on either side of the true depth value (gray point), and (2) and (3) illustrate the cases where both depth values lie on the same side of the true depth value, with one on the right and the other on the left, respectively. In (b), we illustrate the geometric relations and notations used to compute $dis_{ref}^\prime(dis_{pro})$.
• Figure 3: Visual comparisons on LLFF (3 views). Our method produces richer details and more accurate depth than all competitors.
• Figure 4: Visual comparisons on MipNeRF-360 (24 views). In large-scale scenes, NexusGS preserve detailed textures while others struggle.
• Figure 5: Visual comparisons on the DTU dataset (3 views). Our method produces a more comprehensive point cloud than others, resulting in higher-quality renderings.