RoEL: Robust Event-based 3D Line Reconstruction

TL;DR

The results confirm that the proposed line-based formulation is a robust and effective approach for the practical deployment of event-based perceptual modules, and can be flexibly applied to multimodal scenarios.

Abstract

Event cameras in motion tend to detect object boundaries or texture edges, which produce lines of brightness changes, especially in man-made environments. While lines can constitute a robust intermediate representation that is consistently observed, the sparse nature of lines may lead to drastic deterioration with minor estimation errors. Only a few previous works, often accompanied by additional sensors, utilize lines to compensate for the severe domain discrepancies of event sensors along with unpredictable noise characteristics. We propose a method that can stably extract tracks of varying appearances of lines using a clever algorithmic process that observes multiple representations from various time slices of events, compensating for potential adversaries within the event data. We then propose geometric cost functions that can refine the 3D line maps and camera poses, eliminating projective distortions and depth ambiguities. The 3D line maps are highly compact and can be equipped with our proposed cost function, which can be adapted for any observations that can detect and extract line structures or projections of them, including 3D point cloud maps or image observations. We demonstrate that our formulation is powerful enough to exhibit a significant performance boost in event-based mapping and pose refinement across diverse datasets, and can be flexibly applied to multimodal scenarios. Our results confirm that the proposed line-based formulation is a robust and effective approach for the practical deployment of event-based perceptual modules. Project page: https://gwangtak.github.io/roel/

Figures (14)

• Figure 1: We present RoEL, an event-based 3D line reconstruction pipeline that achieves noise-robust reconstruction by leveraging line correspondences. Our 3D line maps not only provide efficient representations for edge-capturing event cameras, but also enable cross-modal applications such as registration and localization.
• Figure 2: Method overview. Our 3D line mapping pipeline, RoEL, consists of two stages: correspondence search and 3D line reconstruction. The first stage takes events and camera poses as input. Through line detection, plane fitting, and matching processes that take into account the characteristics of event data, it finds line correspondences and identifies the events supporting each line. In the second stage, our method triangulates 2D lines to generate initial 3D lines. We further optimize 3D lines with multi-view observations using cost functions defined in 3D space based on the Grassmann distance. Finally, our method outputs 3D line segments that effectively represent the scene.
• Figure 3: Detection-guided space-time plane fitting. Through plane fitting for each line, raw events are distilled into inlier events, and noisy initial lines are refined. In the center box, the black line depicts an initial line which is used for candidate events selection for plane fitting, the red plane indicates the fitted plane, the red line represents the refined line, and the green dots denote all inlier events. Each line is associated with its corresponding events, and the inaccurate 2D lines are refined, as highlighted in the orange circle.
• Figure 4: Difference from reprojection error. The two columns illustrate the same error-calculation scenario consisting of 3D line estimates and a 2D observation. In this scenario, different 3D line estimates project to the same 2D location, leading to identical reprojection errors. Our Grassmann-based cost evaluates geometric consistency directly in 3D.
• Figure 5: Two different cost functions defined for 2D line observations and 2D event observations. The geodesic distance on the Grassmann manifold serve as the basis for cost functions. The left figure illustrates the cost function defined between a 2D line and a 3D line. We use the plane-to-line Grassmann distance between the red plane, which is back-projected from the 2D line, and the blue 3D line. The right figure shows the cost function defined between an event and a 3D line. We use the plane-to-line Grassmann distance between the green line back-projected from the event and the blue plane connecting the 3D line and the camera center.