Exposing the Unseen: Exposure Time Emulation for Offline Benchmarking of Vision Algorithms

TL;DR

This work proposes a new methodology based on an emulator that can generate images at any exposure time of Automatic-Exposure (AE), and shows that using these images acquired at different exposure times, it can emulate realistic images, keeping a Root-Mean-Square Error below 1.78 % compared to ground truth images.

Abstract

Visual Odometry (VO) is one of the fundamental tasks in computer vision for robotics. However, its performance is deeply affected by High Dynamic Range (HDR) scenes, omnipresent outdoor. While new Automatic-Exposure (AE) approaches to mitigate this have appeared, their comparison in a reproducible manner is problematic. This stems from the fact that the behavior of AE depends on the environment, and it affects the image acquisition process. Consequently, AE has traditionally only been benchmarked in an online manner, making the experiments non-reproducible. To solve this, we propose a new methodology based on an emulator that can generate images at any exposure time. It leverages BorealHDR, a unique multi-exposure stereo dataset collected over 10 km, on 55 trajectories with challenging illumination conditions. Moreover, it includes lidar-inertial-based global maps with pose estimation for each image frame as well as Global Navigation Satellite System (GNSS) data, for comparison. We show that using these images acquired at different exposure times, we can emulate realistic images, keeping a Root-Mean-Square Error (RMSE) below 1.78 % compared to ground truth images. To demonstrate the practicality of our approach for offline benchmarking, we compared three state-of-the-art AE algorithms on key elements of Visual Simultaneous Localization And Mapping (VSLAM) pipeline, against four baselines. Consequently, reproducible evaluation of AE is now possible, speeding up the development of future approaches. Our code and dataset are available online at this link: https://github.com/norlab-ulaval/BorealHDR

Figures (6)

• Figure 1: Upper image: Overhead view of a trajectory from BorealHDR dataset, taken in Montmorency Forest, Québec, Canada. The traveled trajectory is shown in purple. Possible emulated exposure times $\Delta t_{e}$ along the trajectory are depicted in orange. Lower image: Acquired brackets (6), with one used to generate $\Delta t_e = 9ms$ from the upper image. The bracket exposure times $\Delta t_b$ are $\{1, 2, 4, 8, 16, 32\}$ ms, increasing the dynamic range of our capture by 30 dB or 5 stops. Blue and red colorized pixels are over-exposed and under-exposed, respectively.
• Figure 2: Picture of the developed backpack for the dataset acquisition. Main components are identified as follows: ① Two Basler a2A1920-51gcPRO cameras, ② Xsens MTI-30 IMU, ③ VLP16 3D lidar, ④ Emlid Reach RS+ GNSS receiver, and ⑤ Ubiquiti UniFi UAP-AC-M wifi antenna.
• Figure 3: Validation of our emulation framework using 1000.0 ground truth images captured with constant illumination, but varying exposure time. Left:RMSE curves showing the emulation error if a single bracket was always selected, in colors, compared to our bracket selection method HigherNoSat, in black. Red symbols correspond to the emulated exposure times displayed on the right. Right: Qualitative comparison of the distributions of five emulated exposure times (columns) from six different bracket images (rows). The markers are placed next to the selected bracket by HigherNoSat. Ground truth distributions are overlaid in gray for each column.
• Figure 4: SIFT features analysis. (a) Uniformity of detected keypoints for an image divided into a $20\times20$ grid. (b) Number of match features between a pair of images for all AE metrics. The gray shading in (a) and (b) is to highlight the state-of-the-art AE methods. (c) Percentage of successful trajectories based on the number of matches detected. If one image contains less matches than $\tau_\text{matches}$, the sequence is marked as not successful.
• Figure 5: Qualitative example illustrating the trajectory backpack_2023-04-21-10-57-59 from our BorealHDR dataset. The lidar-based trajectory (purple) and $M_\text{Zhang}$'s emulated VO result (orange) are illustrated superimposed on the generated lidar 3D map. The higher drift of the VO localization can be highlighted with the loop closure done in this forest environment. Black color represents the ground, and other colors highlight structures in the environment.