LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis

TL;DR

This work introduces LiDAR4D, a dynamic LiDAR NVS framework that operates in a 4D space-time domain to handle sparse, large-scale driving scenes. It combines a 4D hybrid planar-grid representation with a scene-flow prior to ensure temporal coherence and a global ray-drop refinement to produce realistic LiDAR patterns. Through differentiable neural LiDAR fields and a U-Net-based refinement, LiDAR4D achieves geometry-aware, time-consistent reconstructions and superior novel-view synthesis on KITTI-360 and NuScenes, surpassing prior LiDAR-NeRF and explicit methods. The approach offers a scalable path toward high-fidelity LiDAR data generation for autonomous driving and related applications, with potential for future multi-modal extensions.

Abstract

Although neural radiance fields (NeRFs) have achieved triumphs in image novel view synthesis (NVS), LiDAR NVS remains largely unexplored. Previous LiDAR NVS methods employ a simple shift from image NVS methods while ignoring the dynamic nature and the large-scale reconstruction problem of LiDAR point clouds. In light of this, we propose LiDAR4D, a differentiable LiDAR-only framework for novel space-time LiDAR view synthesis. In consideration of the sparsity and large-scale characteristics, we design a 4D hybrid representation combined with multi-planar and grid features to achieve effective reconstruction in a coarse-to-fine manner. Furthermore, we introduce geometric constraints derived from point clouds to improve temporal consistency. For the realistic synthesis of LiDAR point clouds, we incorporate the global optimization of ray-drop probability to preserve cross-region patterns. Extensive experiments on KITTI-360 and NuScenes datasets demonstrate the superiority of our method in accomplishing geometry-aware and time-consistent dynamic reconstruction. Codes are available at https://github.com/ispc-lab/LiDAR4D.

Figures (22)

• Figure 1: Dynamic scenes of LiDAR point clouds in autonomous driving. Large-scale vehicle motion poses a significant challenge for dynamic reconstruction and novel space-time view synthesis. White dots indicate the ego-car trajectory.
• Figure 2: Overview of our proposed LiDAR4D. For large-scale autonomous driving scenarios, we utilize the 4D hybrid representation, which combines low-resolution multi-planar features and high-resolution hash grid features to achieve effective reconstruction. Then, multi-level spatio-temporal features aggregated by flow MLP are fed into neural LiDAR fields for density, intensity and ray-drop probability prediction. Finally, novel space-time view LiDAR point clouds are synthesized via differentiable rendering. Furthermore, we construct geometric constraints derived from point clouds for temporal consistency and the global optimization of ray-drop for generation realism.
• Figure 3: 4D decomposition of hybrid planar-grid representation. Dynamic features can be further aggregated using flow MLP.
• Figure 4: Qualitative comparison for the hybrid representation. Compared to the noisy intensity reconstruction of LiDAR-NeRF and the blurry one of K-Planes, our hybrid representation achieves more precise and smooth results.
• Figure 5: Qualitative comparison for the ray-drop refinement. The point-wise prediction of ray-drop probability by MLP cannot preserve global patterns well. Instead, LiDAR4D drastically improves generation realism via runtime-optimized U-Net.