Physical Priors Augmented Event-Based 3D Reconstruction

TL;DR

This work tackles dense 3D reconstruction from raw event streams by introducing a physically augmented NeRF framework that leverages motion, density, and geometry priors derived from events. A warp-field based prior extraction, a deterministic event-generation model, and a density-guided patch sampling strategy are integrated into NeRF training, along with geometry regularization to improve texture and geometry fidelity. A large, real-world event dataset with 101 objects and groundtruth images/depth maps is constructed to facilitate evaluation and future research. Results show substantial improvements over prior EventNeRF, particularly in realistic, noisy conditions, and the approach converges more quickly, enabling robust 3D reconstruction under challenging lighting and temporal constraints.

Abstract

3D neural implicit representations play a significant component in many robotic applications. However, reconstructing neural radiance fields (NeRF) from realistic event data remains a challenge due to the sparsities and the lack of information when only event streams are available. In this paper, we utilize motion, geometry, and density priors behind event data to impose strong physical constraints to augment NeRF training. The proposed novel pipeline can directly benefit from those priors to reconstruct 3D scenes without additional inputs. Moreover, we present a novel density-guided patch-based sampling strategy for robust and efficient learning, which not only accelerates training procedures but also conduces to expressions of local geometries. More importantly, we establish the first large dataset for event-based 3D reconstruction, which contains 101 objects with various materials and geometries, along with the groundtruth of images and depth maps for all camera viewpoints, which significantly facilitates other research in the related fields. The code and dataset will be publicly available at https://github.com/Mercerai/PAEv3d.

Figures (7)

• Figure 1: Reconstruction results of original NeRF, semi-dense point cloud from event-based approach, EventNeRF, and Ours under extreme overexposure condition. The left figure is the rendering image and the right side is the depth map.
• Figure 2: The whole pipeline of the proposed approach. There are two main branches, i.e. the prior extraction and NeRF rendering branches. The priors are incorporated into the NeRF pipeline at sampling and loss parts.
• Figure 3: Qualitative Comparisons between ours and benchmark on the synthetic dataset.
• Figure 4: Qualitative Comparisons between ours and benchmark on the realistic dataset.
• Figure 5: Qualitative Results of ablations on synthetic data.