Towards Better Robustness: Pose-Free 3D Gaussian Splatting for Arbitrarily Long Videos

TL;DR

Rob-GS tackles the challenge of reconstructing scenes from arbitrarily long casual videos without known camera poses. It introduces two key ideas: adjacent flow-guided pose tracking and Gaussian visibility-based adaptive segmentation, enabling robust pose estimation and memory-efficient 3D Gaussian Splatting on long sequences. The approach yields state-of-the-art rendering quality and faster training across Tanks and Temples, ScanNet, and a self-captured dataset, while maintaining a unified coordinate frame across local segments. The work advances practical deployment of pose-free 3DGS in uncontrolled real-world footage.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a powerful representation due to its efficiency and high-fidelity rendering. 3DGS training requires a known camera pose for each input view, typically obtained by Structure-from-Motion (SfM) pipelines. Pioneering works have attempted to relax this restriction but still face difficulties when handling long sequences with complex camera trajectories. In this paper, we propose Rob-GS, a robust framework to progressively estimate camera poses and optimize 3DGS for arbitrarily long video inputs. In particular, by leveraging the inherent continuity of videos, we design an adjacent pose tracking method to ensure stable pose estimation between consecutive frames. To handle arbitrarily long inputs, we propose a Gaussian visibility retention check strategy to adaptively split the video sequence into several segments and optimize them separately. Extensive experiments on Tanks and Temples, ScanNet, and a self-captured dataset show that Rob-GS outperforms the state-of-the-arts.

Figures (8)

• Figure 1: The overall framework of Rob-GS. We use 3D Gaussians (Sec. \ref{['sec:3dgs']}) as the scene representation, and progressively estimate camera poses using a robust tracking approach (Sec. \ref{['sec:pose']}) that leverages adjacent image pairs. To handle long video sequence, we design an adaptive segmentation scheme (Sec. \ref{['sec:GS optimization']}) to split the video sequence into several local segments and optimize them individually.
• Figure 2: Gaussian visibility retention check strategy. The blue camera exhibits a higher visibility retention rate that exceeds the threshold. Therefore, the blue camera's frame should be added to the current segment, while the red camera's frame should be assigned to a new segment.
• Figure 3: Qualitative comparison for novel view synthesis on Tanks and Temples. Our approach produces sharper details than other baselines.
• Figure 4: Qualitative comparison for novel view synthesis on self-captured dataset. Our approach produces the most realistic rendering results among other baselines. Highly recommend zooming in for better comparison.
• Figure 5: The visualizations of our ablation study. Artifacts will appear when any of our proposed components are removed.