MV-ROPE: Multi-view Constraints for Robust Category-level Object Pose and Size Estimation

TL;DR

This work proposes a novel solution that makes use of RGB video streams that exhibits comparable performance to state-of-the-art RGB-D methods and demonstrates a significant advantage in scenarios where depth input is absent or the quality of depth sensing is limited.

Abstract

Recently there has been a growing interest in category-level object pose and size estimation, and prevailing methods commonly rely on single view RGB-D images. However, one disadvantage of such methods is that they require accurate depth maps which cannot be produced by consumer-grade sensors. Furthermore, many practical real-world situations involve a moving camera that continuously observes its surroundings, and the temporal information of the input video streams is simply overlooked by single-view methods. We propose a novel solution that makes use of RGB video streams. Our framework consists of three modules: a scale-aware monocular dense SLAM solution, a lightweight object pose predictor, and an object-level pose graph optimizer. The SLAM module utilizes a video stream and additional scale-sensitive readings to estimate camera poses and metric depth. The object pose predictor then generates canonical object representations from RGB images. The object pose is estimated through geometric registration of these canonical object representations with estimated object depth points. All per-view estimates finally undergo optimization within a pose graph, culminating in the output of robust and accurate canonical object poses. Our experimental results demonstrate that when utilizing public dataset sequences with high-quality depth information, the proposed method exhibits comparable performance to state-of-the-art RGB-D methods. We also collect and evaluate on new datasets containing depth maps of varying quality to further quantitatively benchmark the proposed method alongside previous RGB-D based methods. We demonstrate a significant advantage in scenarios where depth input is absent or the quality of depth sensing is limited.

Figures (6)

• Figure 1: Overview of the complete, proposed system. The input for the first block (in purple), is a continuous image stream accompanied by scale information. The output of this block includes keyframe camera poses and dense metric depth maps. The second block (in orange) is the single-view object pose estimation module. It utilizes an object pose estimator to obtain NOCS maps for each segmented object. These NOCS maps are then aligned with partial dense depth maps using a RANSAC framework to estimate the pose of each individual object. A third block (in green) finally establishes object correspondences and optimizes object poses over time.
• Figure 2: Graphical models of our scale-aware dense SLAM system. From left to right: model using stereo images, IMU measurements, and depth readings. The red lines represent scale-aware factors in our bundle adjustment layer.
• Figure 3: Illustration of object point clouds captured by different sensors.
• Figure 4: Qualitative results of our method obtained on all 6 sequences of the REAL test dataset wang2019normalized.
• Figure 5: The AP curve of our approach, with the vertical axis representing AP and the horizontal axis representing IoU, rotation error, and translation error of each category.