LBurst: Learning-Based Robotic Burst Feature Extraction for 3D Reconstruction in Low Light

TL;DR

This paper presents a learning architecture for improving 3D reconstructions in low-light conditions by finding features in a burst by detecting and describing high quality true features and less spurious features in low signal-to-noise ratio images.

Abstract

Drones have revolutionized the fields of aerial imaging, mapping, and disaster recovery. However, the deployment of drones in low-light conditions is constrained by the image quality produced by their on-board cameras. In this paper, we present a learning architecture for improving 3D reconstructions in low-light conditions by finding features in a burst. Our approach enhances visual reconstruction by detecting and describing high quality true features and less spurious features in low signal-to-noise ratio images. We demonstrate that our method is capable of handling challenging scenes in millilux illumination, making it a significant step towards drones operating at night and in extremely low-light applications such as underground mining and search and rescue operations.

Figures (8)

• Figure 1: Feature matching at night: A commercial drone (top-left) captures imagery (top-right) in millilux illumination at night. This imagery is too noisy for conventional 3D reconstruction due to the high rate of spurious features detected by conventional methods like SIFT (bottom, blue) and less reliable matches provided by learned feature extractors like R2D2 (bottom, red) on noisy images. The proposed LBurst yields higher-quality true features from reliable regions of the images (bottom, yellow) for reconstruction. In this work, we demonstrate that our feature finder facilitates 3D reconstruction in millilux illumination for existing commercial drones.
• Figure 2: LBurst architecture employs two bursts of images and a flow map with the known ground-truth transformations between the common images of a pair of robotic bursts during training. A specialized burst layer handles burst image processing. The architecture employs a fully convolutional L2-Net, similar to R2D2, generating a 128D descriptor with L2 normalization and a squared element-wise operation yielding ${\mathit{C}_\mathit{feature}}$. Following a 1 x 1 convolution for dimension reduction, a softmax operation computes associated confidence maps.
• Figure 3: Overview of the synthetic robotic burst generation for model training. Cropping and transforming a single image to yield a pair of perspective-altered scenes, and by applying uniform 2D translations within each burst we generate a pair of burst images. We introduce random Gaussian noise to each image in a burst and employ flow maps with known transformations to ensure an accurate vector field between common images within the generated bursts.
• Figure 4: Mean matching performance for scenes with varying viewpoint and illumination. Operating R2D2 on the original HPatches dataset with high SNR (black) acts as a baseline. Operating R2D2 on the generated HPatches robotic burst common images fails to produce accurate matches for moderate (orange) and strong (red) noise. Our method outperforms R2D2 in producing improved feature matches in both moderate (cyan) and strong (blue) noise.
• Figure 5: Low-light scene capture using two drones. (a) (Left) DJI Phantom 4 Pro used for burst capture at 0.125 ms exposure in millilux illumination; (right) Common image from the captured burst. (b) (Left) DJI Mini 3 Pro used for burst capture at the same conditions; (right) Common image from the captured burst. The complete dataset comprises 10 scenes with 15 bursts of 5 images each.