Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras

TL;DR

Photo-SLAM tackles the challenge of achieving real-time, photorealistic mapping in SLAM by proposing a hyper primitives map that unifies explicit geometric features for localization with implicit texture representations. A Gaussian-Pyramid-Based Learning scheme enables progressive, multi-scale feature optimization, while a differentiable 3D Gaussian splatting renderer provides efficient online view synthesis. The approach yields state-of-the-art performance in online photorealistic mapping across monocular, stereo, and RGB-D settings, with real-time capabilities on embedded hardware such as Jetson devices. This hybrid explicit-implicit framework offers practical robustness and rendering quality for robotics applications in unknown environments.

Abstract

The integration of neural rendering and the SLAM system recently showed promising results in joint localization and photorealistic view reconstruction. However, existing methods, fully relying on implicit representations, are so resource-hungry that they cannot run on portable devices, which deviates from the original intention of SLAM. In this paper, we present Photo-SLAM, a novel SLAM framework with a hyper primitives map. Specifically, we simultaneously exploit explicit geometric features for localization and learn implicit photometric features to represent the texture information of the observed environment. In addition to actively densifying hyper primitives based on geometric features, we further introduce a Gaussian-Pyramid-based training method to progressively learn multi-level features, enhancing photorealistic mapping performance. The extensive experiments with monocular, stereo, and RGB-D datasets prove that our proposed system Photo-SLAM significantly outperforms current state-of-the-art SLAM systems for online photorealistic mapping, e.g., PSNR is 30% higher and rendering speed is hundreds of times faster in the Replica dataset. Moreover, the Photo-SLAM can run at real-time speed using an embedded platform such as Jetson AGX Orin, showing the potential of robotics applications.

Figures (16)

• Figure 1: Rendering and trajectory results. Photo-SLAM can reconstruct high-fidelity views of scenes using monocular, stereo, and RGB-D cameras while render speed is up to 1000 FPS.
• Figure 2: The Photo-SLAM contains four main components, including localization, explicit geometry mapping, implicit photorealistic mapping, and loop closure components, while maintaining a map with hyper primitives.
• Figure 3: Comparison of different progressive training methods. The encoder $\mathcal{E}_n$ here represents a structure to regress features $\mathcal{F}_n$ which can be an MLP, voxel grid, hash table, positional encoding, etc. The decoder $\mathcal{D}_n$ here represents a structure converting $\mathcal{F}_n$ into density, color, or other information. We proposed a new method based on the Gaussian pyramid to efficiently learn multi-level features.
• Figure 4: We make use of initial geometric information to densify hyper primitives.
• Figure 5: Training process based on the Gaussian pyramid.