PointSLAM++: Robust Dense Neural Gaussian Point Cloud-based SLAM

TL;DR

PointSLAM++ tackles depth-noise sensitivity and view-dependent appearance in real-time RGB-D SLAM by integrating progressive pose optimization with a hierarchical neural Gaussian representation. It introduces anchor-point neural Gaussians, a two-tier structure of primary and secondary anchors, and a view-direction embedding to enable robust, photorealistic mapping and rendering. The method demonstrates superior reconstruction fidelity and tracking stability across Replica, TUM-RGBD, and ScanNet++ datasets, outperforming state-of-the-art baselines and several Gaussian-based rivals, while maintaining real-time capabilities. Its key contributions—PPO, Neural Gaussian anchors with density-adaptive optimization, and view-dependent compensation—collectively improve pose estimation, geometry accuracy, and cross-view rendering, with potential impact on robotics, AR/VR, and intelligent interaction. The work also discusses ablations showing the necessity of PPO and VDC, and acknowledges increased computation as a trade-off for higher fidelity, suggesting directions for efficiency-focused future work.

Abstract

Real-time 3D reconstruction is crucial for robotics and augmented reality, yet current simultaneous localization and mapping(SLAM) approaches often struggle to maintain structural consistency and robust pose estimation in the presence of depth noise. This work introduces PointSLAM++, a novel RGB-D SLAM system that leverages a hierarchically constrained neural Gaussian representation to preserve structural relationships while generating Gaussian primitives for scene mapping. It also employs progressive pose optimization to mitigate depth sensor noise, significantly enhancing localization accuracy. Furthermore, it utilizes a dynamic neural representation graph that adjusts the distribution of Gaussian nodes based on local geometric complexity, enabling the map to adapt to intricate scene details in real time. This combination yields high-precision 3D mapping and photorealistic scene rendering. Experimental results show PointSLAM++ outperforms existing 3DGS-based SLAM methods in reconstruction accuracy and rendering quality, demonstrating its advantages for large-scale AR and robotics.

Figures (5)

• Figure 1: PointSLAM++ outperforms state-of-the-art methods from CVPR 2024 in photorealistic 3D reconstruction and camera tracking, excelling in key metrics. In complex environments where MonoGS fails, it maintains accurate localization and high-quality 3D mapping.
• Figure 2: SLAM System Overview. The system’s input is a sequence of RGB-D frames. We generate a point cloud by downsampling and reprojecting the current depth image, and we estimate the current pose using GICP and ORB features. During tracking, we create anchor points from the point cloud and utilize a neural network to predict the Gaussian distribution in the scene, rendering the scene using a specialized Gaussian rasterizer. We continuously optimize the multi-layer perceptron (MLP) during the mapping process using the RGB-D information. Right: We use anchor points and features as input to the MLP to predict the various attributes of the Gaussians.
• Figure 3: GICP Mismatches in TUM Scenes. Even with high-precision depth data, real-world noise can lead to misalignments in GICP-based pose estimation, as illustrated here in the TUM dataset.
• Figure 4: Hierarchical Anchor Point Optimization. Primary anchor points (red) are derived from ORB features and remain stable throughout SLAM, while secondary anchor points (yellow) are introduced or culled based on voxel-wise gradient checks (purple) of neural Gaussian points.
• Figure 5: Comparison of Rendering Results. We present scenes from three datasets: Room0 (Replica), fr3 (TUM-RGBD), and bfd3fd54d2 (ScanNet++). Our method achieves rendering results that closely match the ground truth, demonstrating high accuracy. Additional results are available in the supplementary materials.