SaENeRF: Suppressing Artifacts in Event-based Neural Radiance Fields

TL;DR

SaENeRF tackles artifacts in event-based NeRF reconstructions by introducing an event normalization loss that aligns predicted log-radiance changes with accumulated event polarities and a zero-events regularization framework to suppress artifact-prone regions, enabling 3D-consistent dense photorealistic reconstructions from moving-event streams. It employs Temporal Average of Predicted Event Thresholds (TAoPET) and Proportion of Appropriate Pixels (PoAP) within a multi-window, self-supervised optimization built on an Event Generation Model, while utilizing the NeRF representation $F_ heta:(oldsymbol{x},oldsymbol{d})\rightarrow(oldsymbol{ ho},oldsymbol{c})$ and standard volume rendering. Compared to recent baselines (EventNeRF, E-NeRF, E2VID+NeRF), SaENeRF achieves significantly reduced artifacts and improved novel-view synthesis on synthetic and real data, though still relies on known poses. Limitations include residual artifacts and the requirement for pose information, pointing to future directions in pose-unknown event-based implicit SLAM and further artifact-elimination strategies grounded in events.

Abstract

Event cameras are neuromorphic vision sensors that asynchronously capture changes in logarithmic brightness changes, offering significant advantages such as low latency, low power consumption, low bandwidth, and high dynamic range. While these characteristics make them ideal for high-speed scenarios, reconstructing geometrically consistent and photometrically accurate 3D representations from event data remains fundamentally challenging. Current event-based Neural Radiance Fields (NeRF) methods partially address these challenges but suffer from persistent artifacts caused by aggressive network learning in early stages and the inherent noise of event cameras. To overcome these limitations, we present SaENeRF, a novel self-supervised framework that effectively suppresses artifacts and enables 3D-consistent, dense, and photorealistic NeRF reconstruction of static scenes solely from event streams. Our approach normalizes predicted radiance variations based on accumulated event polarities, facilitating progressive and rapid learning for scene representation construction. Additionally, we introduce regularization losses specifically designed to suppress artifacts in regions where photometric changes fall below the event threshold and simultaneously enhance the light intensity difference of non-zero events, thereby improving the visual fidelity of the reconstructed scene. Extensive qualitative and quantitative experiments demonstrate that our method significantly reduces artifacts and achieves superior reconstruction quality compared to existing methods. The code is available at https://github.com/Mr-firework/SaENeRF.

Figures (7)

• Figure 1: The overview presents our progressive learning method. By normalizing predicted radiance variations based on accumulated event polarities, our approach facilitates progressive and rapid training. Furthermore, recognizing the inconsistency of artifacts across multiple views, we incorporate zero-event regularization losses to suppress artifacts in zero accumulated event polarities, thereby enhancing the overall outcome.
• Figure 2: EventNeRF adopts an aggressive joint optimization strategy, simultaneously learning in geometric structure via PoAP and light difference via TAoPET, but often introduces artifacts during early training phases. In contrast, E-NeRF† and SaENeRF match EventNeRF's geometric recovery speed while adopting a more cautious light difference learning approach. However, E-NeRF†'s no-event loss klenk2023e-nerf leads to reduced contrast and incomplete artifact suppression.
• Figure 3: Comparison of our method with EventNeRF, E2VID+NeRF, E-NeRF and its reimplemented real-time versions in synthetic sequences trained using only event input (no RGB).
• Figure 4: Comparison of our method with EventNeRF and E-NeRF†, presented in rendered RGB and Depth Maps in the different real sequences.
• Figure 5: Comparison of SaENeRF NGP against EventNeRF NGP* and E-NeRF* trained for only 5000 iterations in real sequences.