PoseStreamer: A Multi-modal Framework for 6DoF Pose Estimation of Unseen Moving Objects

TL;DR

Unseen moving objects under high-speed motion pose challenges for traditional RGB-based 6DoF pose estimation due to motion blur. PoseStreamer addresses this by fusing RGB with event-based modalities through three components: AMQ for temporal orientation consistency, M3D for robust 3D center priors from SD/TD cues, and RPF for geometric refinement along camera rays. The paper introduces MoCapCube6D, a multi-modal dataset for benchmarking high-speed 6DoF pose estimation, and demonstrates superior accuracy and generalization to unseen objects with a template-free framework. The approach achieves competitive runtime, enabling practical deployment in dynamic environments.

Abstract

Six degree of freedom (6DoF) pose estimation for novel objects is a critical task in computer vision, yet it faces significant challenges in high-speed and low-light scenarios where standard RGB cameras suffer from motion blur. While event cameras offer a promising solution due to their high temporal resolution, current 6DoF pose estimation methods typically yield suboptimal performance in high-speed object moving scenarios. To address this gap, we propose PoseStreamer, a robust multi-modal 6DoF pose estimation framework designed specifically on high-speed moving scenarios. Our approach integrates three core components: an Adaptive Pose Memory Queue that utilizes historical orientation cues for temporal consistency, an Object-centric 2D Tracker that provides strong 2D priors to boost 3D center recall, and a Ray Pose Filter for geometric refinement along camera rays. Furthermore, we introduce MoCapCube6D, a novel multi-modal dataset constructed to benchmark performance under rapid motion. Extensive experiments demonstrate that PoseStreamer not only achieves superior accuracy in high-speed moving scenarios, but also exhibits strong generalizability as a template-free framework for unseen moving objects.

Figures (3)

• Figure 1: Overview of the PoseStreamer Architecture. The framework proceeds in three stages: (A) Unseen object initialization via RGB-based reconstruction and the Adaptive Pose Memory Queue (AMQ). (B) High-speed 3D center estimation via the Multi-modality 3D Tracker (M3D) on SD and TD streams. (C) Fine-grained 6DoF optimization via the Ray Pose Filter (RPF). The filter samples and selects pose hypotheses along the camera ray.
• Figure 2: Details of Multi-modality 3D tracker. Features $F_L$ and $F_R$ are extracted from the left and right Tianmouc cameras, respectively, with modalities including RGB, TD and SD. Feature points are clustered into groups according to motion consistency, after which the 2D centroids $C_{2D}^L$ and $C_{2D}^R$ are computed from the left and right views. Finally, the 3D object center $\mathbf{C}$ is obtained via disparity-based stereo triangulation.
• Figure 3: Distribution comparison. showing that the uniform distribution is bounded between -1 and 1, unlike the Laplace and Gaussian distributions, which are unbounded.