Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video

TL;DR

This work critiques the reliance on noise-free data for ego-motion and 3D reconstruction, introducing a scalable framework to synthesize realistic RGB-D perturbations and a large Robust-Ego3D benchmark to expose robustness gaps. It proposes CorrGS, a robust learning system that combines correspondence-guided pose learning (CPL) with appearance restoration (CARL) within Gaussian Splatting, enabling stable, photorealistic 3D maps from noisy, sparse-view video. Through extensive synthetic and real-world pilots, CorrGS shows superior ego-motion accuracy and denoised 3D reconstruction, particularly under fast motion and dynamic illumination, highlighting the practical impact for robust perception in autonomous systems. Overall, the paper provides a scalable benchmark and a robust learning framework that bridges the gap between idealized datasets and real-world deployment of dense Neural SLAM.

Abstract

We aim to redefine robust ego-motion estimation and photorealistic 3D reconstruction by addressing a critical limitation: the reliance on noise-free data in existing models. While such sanitized conditions simplify evaluation, they fail to capture the unpredictable, noisy complexities of real-world environments. Dynamic motion, sensor imperfections, and synchronization perturbations lead to sharp performance declines when these models are deployed in practice, revealing an urgent need for frameworks that embrace and excel under real-world noise. To bridge this gap, we tackle three core challenges: scalable data generation, comprehensive benchmarking, and model robustness enhancement. First, we introduce a scalable noisy data synthesis pipeline that generates diverse datasets simulating complex motion, sensor imperfections, and synchronization errors. Second, we leverage this pipeline to create Robust-Ego3D, a benchmark rigorously designed to expose noise-induced performance degradation, highlighting the limitations of current learning-based methods in ego-motion accuracy and 3D reconstruction quality. Third, we propose Correspondence-guided Gaussian Splatting (CorrGS), a novel test-time adaptation method that progressively refines an internal clean 3D representation by aligning noisy observations with rendered RGB-D frames from clean 3D map, enhancing geometric alignment and appearance restoration through visual correspondence. Extensive experiments on synthetic and real-world data demonstrate that CorrGS consistently outperforms prior state-of-the-art methods, particularly in scenarios involving rapid motion and dynamic illumination.

Figures (63)

• Figure 1: Towards robust ego-motion and photorealistic 3D reconstruction. (a) Previous approaches rely on synthetic datasets with perfect conditions (noise-free and smooth motion). (b) Real-world data introduce inherent noise and complexities. We present a customizable noisy data synthesis pipeline to evaluate methods under realistic noisy conditions. (c) Our Robust-Ego3D benchmark reveals that existing methods produce noisy 3D reconstructions from noisy, sparse-view videos, whereas (d) our proposed CorrGS achieves photorealistic, noise-free 3D reconstruction.
• Figure 2: Framework for noisy data synthesis, model estimation, and evaluation.
• Figure 3: Perturbation taxonomy for RGB-D sensing.
• Figure 4: Noisy data synthesis pipeline.
• Figure 5: Effect of Shot Noise on 3D reconstruction of Nice-SLAM.

Theorems & Definitions (12)

• Theorem A
• proof
• Theorem B
• proof
• Theorem C
• proof
• Theorem D
• proof
• Theorem E
• proof