SN-LiDAR: Semantic Neural Fields for Novel Space-time View LiDAR Synthesis

TL;DR

SN-LiDAR tackles semantic LiDAR novel view synthesis by introducing semantic neural LiDAR fields that jointly optimize geometry and semantics. It blends a global planar-grid geometric representation with a local CNN-based semantic encoder, feeding into differentiable rendering to produce depth, intensity, semantics, and ray-drop for novel space-time LiDAR views. The method demonstrates superior semantic reconstruction and robust geometric results on SemanticKITTI and KITTI-360, particularly for dynamic objects and large-scale scenes. This framework enables semantic-aware LiDAR synthesis without reliance on extensive pre-trained LiDAR semantic models, with potential benefits for autonomous driving perception and data augmentation.

Abstract

Recent research has begun exploring novel view synthesis (NVS) for LiDAR point clouds, aiming to generate realistic LiDAR scans from unseen viewpoints. However, most existing approaches do not reconstruct semantic labels, which are crucial for many downstream applications such as autonomous driving and robotic perception. Unlike images, which benefit from powerful segmentation models, LiDAR point clouds lack such large-scale pre-trained models, making semantic annotation time-consuming and labor-intensive. To address this challenge, we propose SN-LiDAR, a method that jointly performs accurate semantic segmentation, high-quality geometric reconstruction, and realistic LiDAR synthesis. Specifically, we employ a coarse-to-fine planar-grid feature representation to extract global features from multi-frame point clouds and leverage a CNN-based encoder to extract local semantic features from the current frame point cloud. Extensive experiments on SemanticKITTI and KITTI-360 demonstrate the superiority of SN-LiDAR in both semantic and geometric reconstruction, effectively handling dynamic objects and large-scale scenes. Codes will be available on https://github.com/dtc111111/SN-Lidar.

Figures (8)

• Figure 1: Novel space-time view LiDAR Synthesis with semantics in autonomous driving. Large-scale scenes and dynamic objects are main challenges.
• Figure 2: Overall architecture of our proposed SN-LiDAR. For large-scale sparse point clouds in autonomous driving, we combine global geometric and local semantic features within our local-to-global feature representation. The features are fed into semantic neural LiDAR fields for density, intensity, semantic and ray-drop probability prediction. Finally, novel space-time view LiDAR semantic point clouds are synthesized through differentiable rendering and back projecting.
• Figure 3: Local CNN-based semantic encoder. It extracts semantic features for 1-channel range images.
• Figure 4: Qualitative comparison for LiDAR point cloud reconstruction and synthesis on SemanticKITTI. The white box shows the point cloud of the pedestrian.
• Figure 5: Qualitative comparison for LiDAR semantic reconstruction and synthesis on SemanticKITTI. The gray box displays the semantic label of the cyclist, and the red box shows the semantic label of the pedestrian, with an enlarged view on the left.