Neural Active Structure-from-Motion in Dark and Textureless Environment

TL;DR

A full optimization framework of the volumetric shape that employs neural signed distance fields (Neural-SDF) for SL with the goal of not only reconstructing the scene shape but also estimating the poses for each motion of the system.

Abstract

Active 3D measurement, especially structured light (SL) has been widely used in various fields for its robustness against textureless or equivalent surfaces by low light illumination. In addition, reconstruction of large scenes by moving the SL system has become popular, however, there have been few practical techniques to obtain the system's precise pose information only from images, since most conventional techniques are based on image features, which cannot be retrieved under textureless environments. In this paper, we propose a simultaneous shape reconstruction and pose estimation technique for SL systems from an image set where sparsely projected patterns onto the scene are observed (i.e. no scene texture information), which we call Active SfM. To achieve this, we propose a full optimization framework of the volumetric shape that employs neural signed distance fields (Neural-SDF) for SL with the goal of not only reconstructing the scene shape but also estimating the poses for each motion of the system. Experimental results show that the proposed method is able to achieve accurate shape reconstruction as well as pose estimation from images where only projected patterns are observed.

Figures (10)

• Figure 1: Concept of Active Structure-from-Motion (Active SfM). The system consists of a camera and projectors. Images are captured in an extremely dark environment, where texture information is missing. Our goal is to recover the scene shape and the system poses with unreliable initialization from the projected patterns.
• Figure 2: Left: A failure case of ICP-based pose estimation with NeRF-Synthetic (Lego) scene with cross laser projectors. Green arrows: Ground-truth camera poses. Red arrows: Estimated camera poses via ICP with sparsely reconstructed point clouds. Note that the Ground-truth poses are used for initialization. Right: A failure case of SuperGlue sarlin20superglue feature matching with NeRF-Synthetic (Lego) scene with little illumination (No matches are detected). Note that the contrast is enhanced for visualization.
• Figure 3: Pipeline schematics of the proposed method. Parameters marked in red are optimized during training.
• Figure 4: System configuration of SL system in the experiments. Note that the lasers are colored red and green in the left figure to make the configuration understood easily, however, single color is sufficient as can be seen in the real system, all green (right).
• Figure 5: Example images of the synthetic data with pattern projection. Left: NeRF-Synthetic. Right: BlendedMVS.