Event-aided Direct Sparse Odometry

TL;DR

EDS is the first method to perform 6-DOF VO using events and frames with a direct approach and outperform all previous event-based odometry solutions.

Abstract

We introduce EDS, a direct monocular visual odometry using events and frames. Our algorithm leverages the event generation model to track the camera motion in the blind time between frames. The method formulates a direct probabilistic approach of observed brightness increments. Per-pixel brightness increments are predicted using a sparse number of selected 3D points and are compared to the events via the brightness increment error to estimate camera motion. The method recovers a semi-dense 3D map using photometric bundle adjustment. EDS is the first method to perform 6-DOF VO using events and frames with a direct approach. By design, it overcomes the problem of changing appearance in indirect methods. We also show that, for a target error performance, EDS can work at lower frame rates than state-of-the-art frame-based VO solutions. This opens the door to low-power motion-tracking applications where frames are sparingly triggered "on demand" and our method tracks the motion in between. We release code and datasets to the public.

Figures (14)

• Figure 1: Camera trajectory and estimated 3D map (left). The top-right inset shows the sliding window map (grayscale points) with the current keyframe map (pseudo-colored blue-red points, according to event polarity). The bottom-right insets show the color image (frame) with the real events (left) and the image obtained by the event generative model (right).
• Figure 2: Block diagram of the proposed event-based direct odometry approach. Events and frames acquired by a camera, such as the DAVIS346 Taverni18tcsii, are fed to the front-end, where they are fused using the event generation model (EGM). The front-end selects sparse points on scene edges (i.e., events) with respect to the keyframes. As the camera moves, it generates events, and the camera pose is estimated with respect to the last keyframe. Poses and keyframes are passed to the back-end, which performs non-linear refinement of the poses and depth estimates via photometric bundle adjustment. These are later fed back to the front-end to sustain the good performance of the VO system. Events are colored in blue/red according to polarity $p_k$, indicating positive/negative brightness increments.
• Figure 3: Event frames \ref{['eq:brightnessIncrementEvents']} (left) and EGM frames \ref{['eq:brightnessIncrementGrad2']} (right) for the monitor sequence. Here, positive brightness changes are displayed in white, and negative ones in black. Gray color means there is no brightness change. This figure contains animations that can be viewed in Acrobat Reader.
• Figure 4: Qualitative comparison on four test sequences from Zhou18eccv. The first three rows depict pseudo-colored inverse depth maps for each method. EVO's color code is yellow-near blue-far, while DSO and EDS colors are red-near blue-far. The depth range is 17 in all sequences. The 3D point cloud reconstructed by EDS is shown in the last row, with grayscale values from the keyframe.
• Figure 5: Average Absolute Trajectory Error (RMS) for ORB-SLAM MurArtal17tro, ORB-SLAM$^\ast$ (w/o loop closure), DSO Engel17pami, DSO$^\ast$ (w/o recovery-tracking) and EDS (ours) on dataset Zhou18eccv. Errors are computed for EDS every time an event-frame is tracked (equal at any frame rate), whereas for the baseline frame-based methods they are computed only when a frame is received (i.e., according to the frame rate).