AlignPose: Generalizable 6D Pose Estimation via Multi-view Feature-metric Alignment

TL;DR

<p>AlignPose tackles the challenge of generalizable 6D object pose estimation from RGB images in multi-view settings without object-specific training. It first gathers per-view pose candidates from any single-view method, aggregates them in a shared world frame, and then refines a single consistent pose via a multi-view feature-metric loss computed across all views using frozen foundation-model features. The refinement optimizes a single pose $T_{WO}$ by minimizing the sum of per-view losses, optimized with Levenberg-Marquardt, and scores poses by their cross-view alignment quality. Extensive experiments on YCB-V, T-LESS, ITODD-MV, and HouseCat6D under the BOP protocol show substantial improvements over both single-view baselines and prior multi-view RGB methods, including challenging textureless, reflective, and transparent objects, demonstrating strong zero-shot generalization and industrial applicability.

Abstract

Single-view RGB model-based object pose estimation methods achieve strong generalization but are fundamentally limited by depth ambiguity, clutter, and occlusions. Multi-view pose estimation methods have the potential to solve these issues, but existing works rely on precise single-view pose estimates or lack generalization to unseen objects. We address these challenges via the following three contributions. First, we introduce AlignPose, a 6D object pose estimation method that aggregates information from multiple extrinsically calibrated RGB views and does not require any object-specific training or symmetry annotation. Second, the key component of this approach is a new multi-view feature-metric refinement specifically designed for object pose. It optimizes a single, consistent world-frame object pose minimizing the feature discrepancy between on-the-fly rendered object features and observed image features across all views simultaneously. Third, we report extensive experiments on four datasets (YCB-V, T-LESS, ITODD-MV, HouseCat6D) using the BOP benchmark evaluation and show that AlignPose outperforms other published methods, especially on challenging industrial datasets where multiple views are readily available in practice.

Figures (9)

• Figure 1: Our feature-based multi-view pose estimation pipeline. Single-view pose candidates are first generated independently for each view using state-of-the-art pose estimation methods (e.g. labbe2022megaposeornek2024foundpose). During aggregation, candidates are transformed into a common coordinate frame, and non-maximum suppression (NMS) is applied to eliminate redundant detections of the same object. These filtered pose candidates $\bm{T}_{WO}$ are then refined using a new multi-view feature-metric refinement to obtain object poses that are consistent from all views.
• Figure 2: Multi-view feature-metric refinement. This figure illustrates a two-view feature-metric refinement. The two cameras show cropped query features (mapping the first three PCA components to RGB values). The partial multi-color point clouds represent the registered features $\mathcal{F}_{CO}$, shifted towards their corresponding camera for visualization purposes. The projection of registered feature 3D coordinates $\mathbf{x}_i$ is represented by a line color coded with a matching color of the feature value $\mathbf{p}_i$. During refinement, each registered model feature $\mathbf{p}_i$ is compared to the feature value interpolated from query image features $\mathbf{F}_q$ at the location of the corresponding 3D point $\mathbf{x}_i$ projected into the query image at ${\pi}_C \left( \bm{T}_{CO} \mathbf{x}_i \right)$, according to \ref{['eq:LFE']}. For clarity, only a few (3 per view) projected 3D points are shown.
• Figure 3: Qualitative results of multi-view refinement on the YCB-V and HouseCat6D datasets. 1st column: one of the input images, 2nd column: single-view pose estimates in blue, ground-truth shown using textured models. 3rd column: results of CosyPose multi-view baseline. 4th column: results of our multi-view method. CosyPose and our method both use single-view pose candidates from four input views. Please see the major improvements by our method: "clamp" (top row), "scissors" (2nd row), "glass" (3rd row), "knife" (4th row).
• Figure 4: Qualitative results of multi-view refinement on the T-LESS dataset. 1st column: one of the four input images. 2nd column: single-view pose estimates obtained by MegaPose labbe2022megapose in blue; ground-truth poses in red. 3rd column: results of CosyPose multi-view baseline. 4th column: results of our multi-view pose estimation method. CosyPose and our method both use MegaPose single-view pose candidates from four input views/images. In this challenging setting with multiple textureless objects our approach outputs significantly better poses than CosyPose (see rows 1-2) and is able to correctly obtain poses for more objects (see rows 1-4).
• Figure 5: Non Maximum Suppression ablation for the YCB-V dataset. We perform NMS with different geometric representations of objects (AA: axis-aligned boxes; O: oriented boxes; CH: convex hulls; S: bounding spheres).