An Illumination-Robust Feature Extractor Augmented by Relightable 3D Reconstruction

TL;DR

A self-supervised framework is proposed for extracting features with advantages in repeatability for key points and similarity for descriptors across good and bad illumination conditions, where the recently developed relightable 3D reconstruction techniques are adopted for rapid and direct data generation with varying illumination conditions.

Abstract

Visual features, whose description often relies on the local intensity and gradient direction, have found wide applications in robot navigation and localization in recent years. However, the extraction of visual features is usually disturbed by the variation of illumination conditions, making it challenging for real-world applications. Previous works have addressed this issue by establishing datasets with variations in illumination conditions, but can be costly and time-consuming. This paper proposes a design procedure for an illumination-robust feature extractor, where the recently developed relightable 3D reconstruction techniques are adopted for rapid and direct data generation with varying illumination conditions. A self-supervised framework is proposed for extracting features with advantages in repeatability for key points and similarity for descriptors across good and bad illumination conditions. Experiments are conducted to demonstrate the effectiveness of the proposed method for robust feature extraction. Ablation studies also indicate the effectiveness of the self-supervised framework design.

Figures (4)

• Figure 1: Feature extraction performance tends to degrade with worse illumination conditions (right column compared to left, where red point indicates the missing features). The illumination-robust feature extractor can alleviate such degradation. As an example, the proposed method in (b) alleviates the drain of feature points (repeatability improves from 50.85% to 72.02%) and mismatching of descriptors (map of descriptor improves from 0.719 to 0.853) compared with the SuperPoint method in (a).
• Figure 2: Training pipeline of the proposed illumination-robust feature extractor.
• Figure 3: The proposed data augmentation method can generate figures with (a) different scene configurations, (b) different camera views, and (c) different illumination conditions, for the training of the illumination-robust feature extractor.
• Figure 4: The feature extraction results of the proposed illumination-robust feature extractor across different illumination conditions.