Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera

Zibin Liu, Banglei Guan, Yang Shang, Qifeng Yu, Laurent Kneip

TL;DR

This work proposes a line-based robust pose estimation and tracking method for planar or non-planar objects using an event camera, and devised and established an event-based moving object dataset.

Abstract

Pose estimation and tracking of objects is a fundamental application in 3D vision. Event cameras possess remarkable attributes such as high dynamic range, low latency, and resilience against motion blur, which enables them to address challenging high dynamic range scenes or high-speed motion. These features make event cameras an ideal complement over standard cameras for object pose estimation. In this work, we propose a line-based robust pose estimation and tracking method for planar or non-planar objects using an event camera. Firstly, we extract object lines directly from events, then provide an initial pose using a globally-optimal Branch-and-Bound approach, where 2D-3D line correspondences are not known in advance. Subsequently, we utilize event-line matching to establish correspondences between 2D events and 3D models. Furthermore, object poses are refined and continuously tracked by minimizing event-line distances. Events are assigned different weights based on these distances, employing robust estimation algorithms. To evaluate the precision of the proposed methods in object pose estimation and tracking, we have devised and established an event-based moving object dataset. Compared against state-of-the-art methods, the robustness and accuracy of our methods have been validated both on synthetic experiments and the proposed dataset. The source code is available at https://github.com/Zibin6/LOPET.

Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera

TL;DR

This work proposes a line-based robust pose estimation and tracking method for planar or non-planar objects using an event camera, and devised and established an event-based moving object dataset.

Abstract

Pose estimation and tracking of objects is a fundamental application in 3D vision. Event cameras possess remarkable attributes such as high dynamic range, low latency, and resilience against motion blur, which enables them to address challenging high dynamic range scenes or high-speed motion. These features make event cameras an ideal complement over standard cameras for object pose estimation. In this work, we propose a line-based robust pose estimation and tracking method for planar or non-planar objects using an event camera. Firstly, we extract object lines directly from events, then provide an initial pose using a globally-optimal Branch-and-Bound approach, where 2D-3D line correspondences are not known in advance. Subsequently, we utilize event-line matching to establish correspondences between 2D events and 3D models. Furthermore, object poses are refined and continuously tracked by minimizing event-line distances. Events are assigned different weights based on these distances, employing robust estimation algorithms. To evaluate the precision of the proposed methods in object pose estimation and tracking, we have devised and established an event-based moving object dataset. Compared against state-of-the-art methods, the robustness and accuracy of our methods have been validated both on synthetic experiments and the proposed dataset. The source code is available at https://github.com/Zibin6/LOPET.
Paper Structure (14 sections, 2 theorems, 33 equations, 13 figures, 3 tables, 1 algorithm)

This paper contains 14 sections, 2 theorems, 33 equations, 13 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pose from RGB images
  4. Pose from events
  5. Line-based Object Pose Estimation and Tracking
  6. Event-based Line Detection
  7. Initial Pose Estimation
  8. Event-line Matching
  9. Event-based Pose Optimization and Tracking
  10. Experimental Evaluation
  11. Synthetic Data Experiments
  12. Simulated event experiment
  13. Real Data Experiments
  14. Conclusion

Key Result

Lemma 1

Let ${\mathbf{R}_\mathbf{a_1}}$, ${\mathbf{R}}_{\mathbf{a_2}}$ be rotation matrices, $\mathbf{a_1}$, $\mathbf{a_2}$ their corresponding axis-angle representations. For a vector $\mathbf{v}\in {{\mathbb{R}}^{3}}$: where $\angle \left( \cdot ,\cdot \right)$ denotes the angle between the two rotated vectors, which is less than or equal to the Euclidean distance between their rotations' angle-axis r

Figures (13)

  • Figure 1: Block diagram of the proposed method. The core of the method consists of event-based line detection, initial pose estimation, event-line matching, and pose optimization and tracking. Please note that initialization is performed only once. The input to the system comprises the event stream and object model, and the output consists of object poses and trajectories.
  • Figure 2: Hybrid event clustering strategy. We use a spatio-temporal window of events, containing $N$ events closest to the time $t_i$. The parameters $N$ and $\Delta t$ are adaptively adjusted based on the number of events. Red dots correspond to events, and the bounds of the event cluster are marked in blue.
  • Figure 3: The relative pose between the events and the projected lines. The 3D line is parameterized by its two endpoints $\mathbf P_{1}$ and $\mathbf P_{2}$. The projected line lies at the intersection of the plane $\mathbf P_{1} \mathbf{O_c} \mathbf P_{2}$ and the event plane.
  • Figure 4: Illustration of the event-line matching strategy. Events are selected or rejected based on the threshold $d_a$, $d_t$ and $d_m$.
  • Figure 5: Weights are assigned to events based on event-line distances. (a) The colorbar represents the distance $d$ from the event to the blue line. (b) The colorbar indicates the assigned weight $\omega$ of events.
  • ...and 8 more figures

Theorems & Definitions (3)

  • Lemma 1
  • Lemma 2
  • proof