Learning Fine-Grained Correspondence with Cross-Perspective Perception for Open-Vocabulary 6D Object Pose Estimation

TL;DR

FiCoP tackles open-vocabulary 6D object pose estimation by shifting from unconstrained global matching to a fine-grained, patch-level approach guided by a structural prior. The method combines object-centric preprocessing, cross-perspective perspective learning, and a Patch Correlation Predictor to constrain correspondences to topologically relevant regions, enabling robust pose estimation under large viewpoint changes and clutter. Key contributions include the Patch Correlation Predictor, the Cross-Perspective Global Perception module, and an object-centric disentanglement pipeline built on GroundingDINO and SAM, all leading to state-of-the-art AR on REAL275 and Toyota-Light with strong generalization. The work demonstrates that explicit spatial constraints and multi-view reasoning substantially bridge semantic and geometric gaps in open-world perception, with practical impact for autonomous manipulation of unseen objects driven solely by natural language.

Abstract

Open-vocabulary 6D object pose estimation empowers robots to manipulate arbitrary unseen objects guided solely by natural language. However, a critical limitation of existing approaches is their reliance on unconstrained global matching strategies. In open-world scenarios, trying to match anchor features against the entire query image space introduces excessive ambiguity, as target features are easily confused with background distractors. To resolve this, we propose Fine-grained Correspondence Pose Estimation (FiCoP), a framework that transitions from noise-prone global matching to spatially-constrained patch-level correspondence. Our core innovation lies in leveraging a patch-to-patch correlation matrix as a structural prior to narrowing the matching scope, effectively filtering out irrelevant clutter to prevent it from degrading pose estimation. Firstly, we introduce an object-centric disentanglement preprocessing to isolate the semantic target from environmental noise. Secondly, a Cross-Perspective Global Perception (CPGP) module is proposed to fuse dual-view features, establishing structural consensus through explicit context reasoning. Finally, we design a Patch Correlation Predictor (PCP) that generates a precise block-wise association map, acting as a spatial filter to enforce fine-grained, noise-resilient matching. Experiments on the REAL275 and Toyota-Light datasets demonstrate that FiCoP improves Average Recall by 8.0% and 6.1%, respectively, compared to the state-of-the-art method, highlighting its capability to deliver robust and generalized perception for robotic agents operating in complex, unconstrained open-world environments. The source code will be made publicly available at https://github.com/zjjqinyu/FiCoP.

Figures (9)

• Figure 1: Comparison between our method and previous methods. Previous methods oryonhoryon based on global matching are prone to incorrect matching. Our method gradually refines the matching area through object-centric disentanglement and patch-to-patch correlation priors, aiming to promote more accurate matching.
• Figure 2: The framework of our proposed Fine-grained Correspondence Pose Estimation (FiCoP) model. It consists of two stages: (a) a preprocessing pipeline that utilizes an open-vocabulary object detection model and a SAM model to generate cropped object images and masks; (b) a model forwarding process that takes the preprocessing results as input to generate high-resolution features for anchor and query, as well as patch correlation maps.
• Figure 3: Structure of the Cross-Perspective Global Perception (CPGP) module. This module facilitates information interaction between the anchor and query perspectives.
• Figure 4: Structure of the Patch Correlation Predictor (PCP). A patch correlation map is generated through a carefully designed process to establish fine-grained spatial correspondences between anchor and query.
• Figure 5: Training objectives and inference process. (a) The optimization objective comprises two components: a contrastive loss for feature matching and a classification loss for the patch correlation map. (b) During inference, high-similarity feature pairs are selected from fine-grained regions, and relative poses are computed using the Point DSC algorithm.