MixRI: Mixing Features of Reference Images for Novel Object Pose Estimation

TL;DR

MixRI tackles the problem of CAD-based novel object pose estimation from RGB images with a lightweight network that operates without finetuning. It introduces View-Aggregated Point Matching via a Dual-Attention Based Feature Mixer to fuse multi-view reference features with the query, predicting per-point projections and occlusions, followed by SQ-PnP within RANSAC to recover the $6$DoF pose. The method achieves competitive performance on seven core BOP datasets while using far fewer reference images and a smaller network than prior approaches, and it exhibits faster inference and reduced memory, making it well suited for edge-AI applications. This work demonstrates that strong pose estimation can be obtained through multi-view feature fusion and efficient attention-based fusion, reducing the need for large caches or extensive reference sets.

Abstract

We present MixRI, a lightweight network that solves the CAD-based novel object pose estimation problem in RGB images. It can be instantly applied to a novel object at test time without finetuning. We design our network to meet the demands of real-world applications, emphasizing reduced memory requirements and fast inference time. Unlike existing works that utilize many reference images and have large network parameters, we directly match points based on the multi-view information between the query and reference images with a lightweight network. Thanks to our reference image fusion strategy, we significantly decrease the number of reference images, thus decreasing the time needed to process these images and the memory required to store them. Furthermore, with our lightweight network, our method requires less inference time. Though with fewer reference images, experiments on seven core datasets in the BOP challenge show that our method achieves comparable results with other methods that require more reference images and larger network parameters.

Figures (11)

• Figure 1: Comparison of different methods. The area of each bubble is proportional to the size of the network parameters, and the color indicates the number of reference images used. The detectionnguyen2023cnos stage was removed in all speed evaluations. MixRI achieves competitive results while using fewer reference images, a smaller network, and providing shorter inference time.
• Figure 2: Comparison between our pipeline and previous methods. Unlike existing two-stage methods nguyen2024gigaPoselabbe2022megaposeausserlechner2023zs6dornek2023foundpose that first retrieve the closest reference image from abundant reference images, we directly predict the location of sampled 3D object points on the query image from their projections on all reference images. It constructs the 2D-3D correspondence and can solve the 6DoF pose. Hollow points indicate invisibility, while solid points indicate visibility.
• Figure 3: Overview of our network. Unlike existing methods labbe2022megaposeausserlechner2023zs6dnguyen2024gigaPoseornek2023foundpose, which use two stages to compute the pose, we directly input all reference images into our network without a view selection stage. After obtaining all features of the projection belonging to one 3D object point $\mathbf{p}_k$, we use Dual-Attention Based Feature Mixer to fuse their features with the query image feature. Then, we build the cost volume followed by two separate heads to predicate the projection of $\mathbf{p}_k$ on the query image, including the occlusion flag as well as the coordinate.
• Figure 4: Overview of the Dual-Attention Based Feature Mixer. The mixer consists of three modules: SAP (Self-Attention between Points) , SAF (Self-Attention between Frames), and MARQ (Mix-Attention between Reference & Query). These modules perform attention operations between N points within one reference image, between S frames, and between reference images and the query image, respectively.
• Figure 5: Qualitative results on YCB-V. We present the pose estimation results obtained using MixRI. All the results are visualized in error heatmap tremblay2023diffdope which darker blue indicates lower error with respect to the ground truth pose (legend: 0 cm 5 cm).