ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Images

TL;DR

ESR-NeRF tackles emissive-source reconstruction within NeRF-based inverse rendering using only LDR multi-view images captured with lights on and off. It replaces explicit ray tracing with learnable ray-traced fields, introduces a learnable tone-mapper, and enforces light-transport consistency through Light Transport Segments (LTS) losses, enabling progressive, reflection-aware identification of emissive regions. The method demonstrates accurate emissive-source localization, robust scene editing of illumination, and competitive surface reconstruction on DTU even without emissives, marking a first in NeRF-based inverse rendering for emissive scenes. Together, these contributions enable controllable illumination of complex scenes from simple LDR inputs, with practical implications for augmented reality and material-light interaction modeling.

Abstract

Existing NeRF-based inverse rendering methods suppose that scenes are exclusively illuminated by distant light sources, neglecting the potential influence of emissive sources within a scene. In this work, we confront this limitation using LDR multi-view images captured with emissive sources turned on and off. Two key issues must be addressed: 1) ambiguity arising from the limited dynamic range along with unknown lighting details, and 2) the expensive computational cost in volume rendering to backtrace the paths leading to final object colors. We present a novel approach, ESR-NeRF, leveraging neural networks as learnable functions to represent ray-traced fields. By training networks to satisfy light transport segments, we regulate outgoing radiances, progressively identifying emissive sources while being aware of reflection areas. The results on scenes encompassing emissive sources with various properties demonstrate the superiority of ESR-NeRF in qualitative and quantitative ways. Our approach also extends its applicability to the scenes devoid of emissive sources, achieving lower CD metrics on the DTU dataset.

Figures (33)

• Figure 1: Challenges posed by emissive sources in LDR images. Green, red, and blue in thresholded images respectively show true positives, false negatives, and false positives of source identification. Thresholding values are scaled down divided by 255. The contrast between light on and off pixel values is more pronounced in surroundings than emissive sources. Inaccurate reconstruction of emissive sources disrupts scene editing, causing reflection areas to stay static while only the source colors change.
• Figure 2: The pipeline of emissive source reconstruction. Given LDR images with emissive sources on and off, scene components are reconstructed by synthesizing training images and enforcing LTS requirements. Emissive sources are progressively refined via categorizing training rays into uncertain and certain groups. The scenes can be edited with new lighting conditions using reconstructed emissive sources.
• Figure 3: Reconstructed surfaces with the learnable tone-mapper.
• Figure 4: Left: Image with active emissive sources. Right: Identified emissive sources w/o progressive discovery of reflection areas.
• Figure 5: Illustration of the progressive emissive source reconstruction with reflection awareness. Gray color represents the areas belonging to the certain group, while the red (emissive sources) and orange (their reflections) areas belong to the uncertain group.