GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time

TL;DR

GGRt addresses the challenge of pose-free, real-time generalizable 3D view synthesis by coupling an Iterative Pose Optimization Network with a Generalizable 3D-Gaussians model. It introduces a Gaussians Cache and Deferred Back-propagation to enable high-resolution training and fast inference without camera poses, achieving real-time rendering and strong performance on LLFF, Waymo, and KITTI. The method outperforms pose-free NeRF baselines and approaches pose-based 3D-GS methods, with substantial gains from the caching and deferred optimization strategies. These innovations facilitate practical, scalable rendering in unposed, real-world environments. Overall, GGRt advances the integration of computer vision and graphics by delivering pose-free, fast, and high-quality 3D reconstructions suitable for immersive applications.

Abstract

This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative Pose Optimization Network (IPO-Net) and a Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism, the proposed framework can inherently estimate robust relative pose information from the image observations and thus primarily alleviate the requirement of real camera poses. Moreover, we implement a deferred back-propagation mechanism that enables high-resolution training and inference, overcoming the resolution constraints of previous methods. To enhance the speed and efficiency, we further introduce a progressive Gaussian cache module that dynamically adjusts during training and inference. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at $\ge$ 5 FPS and real-time rendering at $\ge$ 100 FPS. Through extensive experimentation, we demonstrate that our method outperforms existing NeRF-based pose-free techniques in terms of inference speed and effectiveness. It can also approach the real pose-based 3D-GS methods. Our contributions provide a significant leap forward for the integration of computer vision and computer graphics into practical applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open datasets and enabling real-time rendering for immersive experiences.

Figures (9)

• Figure 1: Our proposed GGRt stands for the first pose-free generalizable 3D Gaussian splatting approach, capable of inference at over 5 FPS, and delivering real-time rendering performance.
• Figure 2: An overview of our method, demonstrated by using two continuous training steps given $N$ selected nearby images. In the first training step, reference views are selected from nearby time $r\in \mathcal{N} (t)$, then the IPO-Net estimates the relative poses between reference and target image $\mathbf{T}_{r\rightarrow t}$for 3D-Gaussian predictions. Then $\mathbf{I}^1_r \cdots \mathbf{I}^4_r$ forms three image pairs and is fed into the G-3DG model to predict Gaussians $\mathbf{G}_1\cdots\mathbf{G}_3$ for novel view splatting and store them in Gaussians cache. In the second step, since $\mathbf{I}^2_r \cdots \mathbf{I}^4_r$ are collaboratively used by the last step, we directly query their image ID in the Cache Gaussians and pick up corresponding Gaussian points $\mathbf{G}_2,\mathbf{G}_3$ with newly predicted $\mathbf{G}_4$ for novel view splatting.
• Figure 3: Illustration of deferred back-propagation pipeline of our G-3DG model (left column) and the procedure of our local self-attention module in deferred back-propagation (right column). Details are shown in Sec. \ref{['subsec:defer']}.
• Figure 4: Novel view synthesis qualitative outcomes on the LLFF llff dataset under generalized settings, with significant regions highlighted by red rectangles.
• Figure 5: Qualitative results for novel view synthesis on Waymo waymo_open dataset with generalized settings. Areas of distinction are marked with red rectangles.