ActiveNeRF: Learning Accurate 3D Geometry by Active Pattern Projection

TL;DR

ActiveNeRF addresses the geometry accuracy limitations of NeRF by introducing an actively projected high-frequency pattern via a projector fixed to the camera, enabling stronger geometric regularization. It jointly learns the scene geometry and a learnable active light pattern through a differentiable two-stage rendering framework that combines environment radiance with active radiance modulated by a neural BRDF field, and derives depth from ray-weighted measures with a max-weight strategy. The approach yields state-of-the-art geometry reconstruction on synthetic and real data, outperforming OpenMVS and NeRF2Mesh and delivering realistic active-light renderings with a mean PSNR of $30.84$ across eight scenes. This work demonstrates the practical potential of active illumination to enhance multi-view 3D exposure with self-supervision, reducing dependence on ground-truth depth data and improving robustness for applications in robotics and AR/VR. Future directions include extending to dynamic scenes and handling non-diffuse objects, enabling broader deployment of active sensing-based 3D capture.

Abstract

NeRFs have achieved incredible success in novel view synthesis. However, the accuracy of the implicit geometry is unsatisfactory because the passive static environmental illumination has low spatial frequency and cannot provide enough information for accurate geometry reconstruction. In this work, we propose ActiveNeRF, a 3D geometry reconstruction framework, which improves the geometry quality of NeRF by actively projecting patterns of high spatial frequency onto the scene using a projector which has a constant relative pose to the camera. We design a learnable active pattern rendering pipeline which jointly learns the scene geometry and the active pattern. We find that, by adding the active pattern and imposing its consistency across different views, our proposed method outperforms state of the art geometry reconstruction methods qualitatively and quantitatively in both simulation and real experiments. Code is avaliable at https://github.com/hcp16/active_nerf

Figures (10)

• Figure 1: Different from the original NeRF setting where the environment lighting is static and has low spatial frequency, we use a projector to actively project patterns of high spatial frequency onto the scene. Our geometry reconstruction consists of two stages. The first stage only renders static environment light and outputs a rough geometry. the second stage utilizes the rough geometry to compute the active light intensity at each pixel, and fine-tunes the geometry and the active pattern jointly.
• Figure 2: Architecture overview. The rendering process is divided into two stages. The first stage is similar to NeRF in that it renders images without active light (only environment light) and is supervised by ground truth images. The second stage renders active light onto the image from the first stage. Using the depth computed from the first stage, the second stage queries the BRDF value and active light radiance of a surface 3D point. This stage is supervised by ground truth images that contain an active light pattern. Since the whole rendering process is differentiable, the active light pattern is gradually learned from scratch.
• Figure 3: Visualization of occupancy and weight distribution along a single camera ray. The left figure is depth against occupancy $\sigma$ and the right figure is depth against weight $w$. The vertical red line denotes ground truth depth, the green line denotes weight-max depth, and the yellow line denotes weighted sum depth.
• Figure 4: Visualization of the reconstructed point cloud of synthetic images, colored with chamfer distance from the ground truth point cloud.
• Figure 5: We observe that the active light tensor aligns more closely with ground truth as training progresses.