Optical Flow-Guided 6DoF Object Pose Tracking with an Event Camera

TL;DR

The paper tackles robust 6DoF object pose tracking using an event camera, addressing motion blur and lighting challenges inherent to conventional sensors. It introduces a flow-guided framework that jointly leverages a 2D-3D hybrid feature set (corners from Time Surfaces and edges from projected 3D models), event-based optical flow within a spatio-temporal window, and an LM-based pose refinement to iteratively minimize corner-edge distances. Key contributions include (i) a 2D-3D hybrid feature extraction strategy, (ii) an event-based optical-flow estimation formulated as a probability maximization problem, and (iii) an optical-flow-guided optimization loop for continuous pose tracking. Extensive experiments on simulated and real event data demonstrate superior accuracy and robustness over state-of-the-art event-based methods, especially under occlusion and clutter, highlighting potential for real-time AR, robotics, and navigation applications in dynamic lighting and fast-motion environments.

Abstract

Object pose tracking is one of the pivotal technologies in multimedia, attracting ever-growing attention in recent years. Existing methods employing traditional cameras encounter numerous challenges such as motion blur, sensor noise, partial occlusion, and changing lighting conditions. The emerging bio-inspired sensors, particularly event cameras, possess advantages such as high dynamic range and low latency, which hold the potential to address the aforementioned challenges. In this work, we present an optical flow-guided 6DoF object pose tracking method with an event camera. A 2D-3D hybrid feature extraction strategy is firstly utilized to detect corners and edges from events and object models, which characterizes object motion precisely. Then, we search for the optical flow of corners by maximizing the event-associated probability within a spatio-temporal window, and establish the correlation between corners and edges guided by optical flow. Furthermore, by minimizing the distances between corners and edges, the 6DoF object pose is iteratively optimized to achieve continuous pose tracking. Experimental results of both simulated and real events demonstrate that our methods outperform event-based state-of-the-art methods in terms of both accuracy and robustness.

Figures (6)

• Figure 1: Flowchart of the proposed method. The core algorithm includes event-based feature extraction, corner-edge matching, and pose tracking. The method takes event streams, initial pose and the object model as input, continuously producing the 6DoF poses of the object as output.
• Figure 2: Feature extraction results. (a) Corners are extracted from TS, indicated in red. (b) Edges are extracted from the projected point cloud (blue), represented in black.
• Figure 3: The geometric interpretation of optical flow estimation. The optical flow of corners is computed by establishing correspondences between corners and events within a given spatio-temporal event window.
• Figure 4: Pose tracking results of virtual objects in simulated event experiments. The red edges represent the edges of objects, which are reprojected onto TSs using the estimated pose solved by our methods. (a) Tracking results of objects with straight edges, from top to bottom, corresponding to Sequence 01-06. (b) Tracking results of objects with curved edges, from top to bottom, corresponding to Sequence 07-12.
• Figure 5: Pose tracking results of test objects in real-world experiments. The red represents the projected edges of their 3D models, which are reprojected onto TSs using the estimated poses obtained from our methods. The TSs are shown from top to bottom, corresponding to Sequence 13-17.