Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems

Abstract

This paper presents a unified surface reconstruction and rendering framework for LiDAR-visual systems, integrating Neural Radiance Fields (NeRF) and Neural Distance Fields (NDF) to recover both appearance and structural information from posed images and point clouds. We address the structural visible gap between NeRF and NDF by utilizing a visible-aware occupancy map to classify space into the free, occupied, visible unknown, and background regions. This classification facilitates the recovery of a complete appearance and structure of the scene. We unify the training of the NDF and NeRF using a spatial-varying scale SDF-to-density transformation for levels of detail for both structure and appearance. The proposed method leverages the learned NDF for structure-aware NeRF training by an adaptive sphere tracing sampling strategy for accurate structure rendering. In return, NeRF further refines structural in recovering missing or fuzzy structures in the NDF. Extensive experiments demonstrate the superior quality and versatility of the proposed method across various scenarios. To benefit the community, the codes will be released at \url{https://github.com/hku-mars/M2Mapping}.

Figures (13)

• Figure 1: We show our surface reconstruction results (color indicates the normal direction) and rendering results collected by real-world LiDAR-visual sensor systems under different trajectories (red lines): forward-facing (FF), object-centric (OC), and free-view (Free).
• Figure 1: Illustrations of visible-aware occupancy map and structure-aware sampling based on adaptive sphere tracing. Each circle has a radius equal to the SDF value at the sample point.
• Figure 2: The overall pipeline of the proposed multimodal mapping framework, named M2Mapping. Given a series of posed images and LiDAR point clouds, we first construct the visible-aware occupancy map via ray casting. The neural distance field is trained using the ray distance value from the point cloud. The neural distance field guides the structure-aware sampling process of the neural radiance field and predicts the density of each point. The sample points and direction are encoded as features, and the MLP forwards the concatenated features to infer color. The volume rendering accumulates densities and color for novel-view synthesis. ($\bigoplus$ denotes the concatenation operation)
• Figure 2: The qualitative results of FAST-LIVO2 datasets (scenes from top to bottom are Campus, Sculpture, Culture, and Drive). We show our surface reconstruction results on the left, and the red line indicates the training path and the orange cameras indicates the extrapolation views for the right side's rendering results.
• Figure 3: Illustrations of our visible-aware occupancy map. The LiDAR observations derive a standard occupancy grid map (OGM) with states: free, occupied, and unknown. We further classify the unknown states into visible unknown and invisible unknown via images' pixel ray casting. Note that the visible unknown grids arise from unknown grids which is visible from the camera without any occupied grid in between.