CE-NPBG: Connectivity Enhanced Neural Point-Based Graphics for Novel View Synthesis in Autonomous Driving Scenes

TL;DR

CE-NPBG tackles the appearance-geometry mismatch in large-scale autonomous driving NVS by introducing a connectivity graph that links camera views to a subset of LiDAR points, enabling view-relevant point retrieval for neural point-based rendering. It combines rasterized neural descriptors with a U-Net synthesizer and a multi-scale discriminator, trained jointly but decoupled at inference, and demonstrates compatibility with 3D Gaussian Splatting to boost scalability. Across KITTI360 sequences, CE-NPBG outperforms state-of-the-art baselines (NPBG, NPBG++, READ, DS-NeRF, 3DGS) in PSNR/SSIM/LPIPS, thanks to accurate visible-point retrieval and adversarial refinement. The approach yields substantial practical gains in rendering quality and runtime efficiency for large driving scenes, enabling scalable deployment in real-world autonomous systems; future work may extend to language-conditioned and semantic-aware novel view synthesis.

Abstract

Current point-based approaches encounter limitations in scalability and rendering quality when using large 3D point cloud maps because using them directly for novel view synthesis (NVS) leads to degraded visualizations. We identify the primary issue behind these low-quality renderings as a visibility mismatch between geometry and appearance, stemming from using these two modalities together. To address this problem, we present CE-NPBG, a new approach for novel view synthesis (NVS) in large-scale autonomous driving scenes. Our method is a neural point-based technique that leverages two modalities: posed images (cameras) and synchronized raw 3D point clouds (LiDAR). We first employ a connectivity relationship graph between appearance and geometry, which retrieves points from a large 3D point cloud map observed from the current camera perspective and uses them for rendering. By leveraging this connectivity, our method significantly improves rendering quality and enhances run-time and scalability by using only a small subset of points from the large 3D point cloud map. Our approach associates neural descriptors with the points and uses them to synthesize views. To enhance the encoding of these descriptors and elevate rendering quality, we propose a joint adversarial and point rasterization training. During training, we pair an image-synthesizer network with a multi-resolution discriminator. At inference, we decouple them and use the image-synthesizer to generate novel views. We also integrate our proposal into the recent 3D Gaussian Splatting work to highlight its benefits for improved rendering and scalability.

Figures (5)

• Figure 1: The left side shows the projection of 3D points into the camera, which includes seen (green) and unseen (gray) 3D points. Using these projections to render a view results in artifacts, as shown on the right side. Right: an example showing incoherence between the points projections (geometry) and the reference view (appearance). The resulting renderings are deteriorated. The sample depicts the projection of an unseen building in the current camera view. The building is shown in the arbitrary views.
• Figure 2: Our CE-NPBG: a scene is represented by source images and a point cloud. To render a novel view (yellow camera), a visibility module retrieves relevant 3D points, whose descriptors are rasterized into the camera at various resolutions. A refiner network then maps the rasterized images into the novel view. During training, a multi-resolution discriminator improves quality by classifying the generated images as real or fake. $\mathbf{f_i}$ is the descriptor for point $\mathbf{g_i}$.
• Figure 3: Different proposals to address large point cloud rendering: a) clipping points outside certain threshold, b) clustering of the scene, c) taking closest points based on depth-buffer d) Our CE-NPBG: building connectivity relationship between 3D points and poses/images.
• Figure 4: Rendering results of our method and other state-of-the-art methods.
• Figure 5: Using our proposed connectivity relationship for visibility estimation drastically improves scene reconstruction using 3D Gaussian Splatting 3dgs.