EvMAPPER: High Altitude Orthomapping with Event Cameras

TL;DR

This work introduces the first orthomosaic approach using event cameras, which focuses on addressing high-dynamic range and low-light problems in orthomosaics, and enables map generation even in challenging light conditions.

Abstract

Traditionally, unmanned aerial vehicles (UAVs) rely on CMOS-based cameras to collect images about the world below. One of the most successful applications of UAVs is to generate orthomosaics or orthomaps, in which a series of images are integrated together to develop a larger map. However, the use of CMOS-based cameras with global or rolling shutters mean that orthomaps are vulnerable to challenging light conditions, motion blur, and high-speed motion of independently moving objects under the camera. Event cameras are less sensitive to these issues, as their pixels are able to trigger asynchronously on brightness changes. This work introduces the first orthomosaic approach using event cameras. In contrast to existing methods relying only on CMOS cameras, our approach enables map generation even in challenging light conditions, including direct sunlight and after sunset.

Figures (7)

• Figure 1: Top: The Falcon 4 aerial platform used for high-altitude experiments. The sensor stack, equipped with an IMU, a range sensor, an RGB camera, and an event camera, was mounted at the front. Bottom left: artifacts of CMOS-based cameras in high-altitude photography: the sidewalk is washed out due to high brightness. Bottom right: reconstructed frame with event cameras displaying higher level of detail in challenging light conditions.
• Figure 2: System architecture for data acquisition. The synchronization board generates pulse signals that are used to trigger (Blackfly S) or timestamp sensors (event camera, IMU, GNSS). Data is collected on an onboard computer running ROS 2, and synchronization is performed after the fact. The single-point LiDAR distance sensor is the only sensor that is not hardware synchronized.
• Figure 3: Time synchronization pattern inserted in the synchronization signal. The silence gaps in the signal are predefined, enabling to identify the beginning of the timing sequence for all the sensors.
• Figure 4: EvMAPPER data preprocessing pipeline. Events and RGB images are collected when the is not performing aggressive rotations. Events are reconstructed into frames, and RGB images are remapped to match the event frames. The resulting representations are fed into an off-the-shelf orthomap generation tool.
• Figure 5: Ground truth reconstruction using F3.D.1 and F3.D.2 sequences. Left: Orthophoto. Right: Point cloud.