J-NeuS: Joint field optimization for Neural Surface reconstruction in urban scenes with limited image overlap

TL;DR

J-NeuS addresses the challenge of urban surface reconstruction from driving sequences with limited image overlap by jointly optimizing a volumetric NeRF and a Signed Distance Function (SDF) while explicitly modeling cross-representation uncertainty. The method introduces photometric and geometric uncertainty signals to guide Guided Ray Sampling, enabling the volumetric and surface representations to specialize where they perform best. Adaptive relaxation of geometric regularization prevents over-smoothing in uncertain areas, preserving fine structures essential for autonomous driving tasks. Extensive experiments on KITTI-360, Pandaset, Waymo, and nuScenes show that J-NeuS achieves state-of-the-art or near-state-of-the-art 3D reconstruction quality with efficient mesh extraction, demonstrating strong practical impact for urban scene modeling under challenging observation conditions.

Abstract

Reconstructing the surrounding surface geometry from recorded driving sequences poses a significant challenge due to the limited image overlap and complex topology of urban environments. SoTA neural implicit surface reconstruction methods often struggle in such setting, either failing due to small vision overlap or exhibiting suboptimal performance in accurately reconstructing both the surface and fine structures. To address these limitations, we introduce J-NeuS, a novel hybrid implicit surface reconstruction method for large driving sequences with outward facing camera poses. J-NeuS cross-representation uncertainty estimation to tackle ambiguous geometry caused by limited observations. Our method performs joint optimization of two radiance fields in addition to guided sampling achieving accurate reconstruction of large areas along with fine structures in complex urban scenarios. Extensive evaluation on major driving datasets demonstrates the superiority of our approach in reconstructing large driving sequences with limited image overlap, outperforming concurrent SoTA methods.

Figures (12)

• Figure 1: We introduce J-NeuS, a novel hybrid implicit surface reconstruction method specifically designed for large-scale driving sequences with limited camera overlap. Extensive experiments on four major driving datasets demonstrate the superiority of J-NeuS's mesh (top, form left to right: mesh normals, textured mesh and shaded mesh) over previous state-of-the-art methods (bottom).
• Figure 2: Overview of J-NeuS: we jointly train two implicit model: a volumetric representation $\mathcal{F}_\Theta^v$ and a SDF $\mathcal{F}_\Theta^f$ with a mutual guidance provided through our Guided Ray Sampling strategy. A colorized mesh of the scene $\mathcal{E}$ is periodically extracted from the SDF representation.
• Figure 3: Visualization of $\mu_d$ and $\mu_c$, maximum clamped at 0.3.
• Figure 4: Mean photometric results for each dataset. We highlight best performing methods in green and second one underlined.
• Figure 5: Ablations study: we deactivate some J-NeuS 's key components. (a) without GRS directed by $\mathcal{\mu}_{d,c}$ (b) without adaptive relaxation on Eikonal constrain and normal supervision