Category-Level and Open-Set Object Pose Estimation for Robotics

TL;DR

This paper surveys category-level and open-set 6D object pose estimation methods evaluated on CAMERA and REAL275, focusing on single-frame inference without requiring a 3D model at test time. It analyzes input modalities, network architectures, symmetry handling strategies, and 6D pose solvers, highlighting how each facet affects generalization and the potential to bridge to open-set scenarios. Key findings include the critical role of depth data, the effectiveness of implicit symmetry handling when supported by suitable architectures, and the continued value of deterministic solvers like Umeyama or TEASER++ for geometry-agnostic pose estimation. The work provides actionable recommendations to improve generalization to unseen categories and outlines a roadmap for bridging category-level methods to open-set pose estimation in robotics.

Abstract

Object pose estimation enables a variety of tasks in computer vision and robotics, including scene understanding and robotic grasping. The complexity of a pose estimation task depends on the unknown variables related to the target object. While instance-level methods already excel for opaque and Lambertian objects, category-level and open-set methods, where texture, shape, and size are partially or entirely unknown, still struggle with these basic material properties. Since texture is unknown in these scenarios, it cannot be used for disambiguating object symmetries, another core challenge of 6D object pose estimation. The complexity of estimating 6D poses with such a manifold of unknowns led to various datasets, accuracy metrics, and algorithmic solutions. This paper compares datasets, accuracy metrics, and algorithms for solving 6D pose estimation on the category-level. Based on this comparison, we analyze how to bridge category-level and open-set object pose estimation to reach generalization and provide actionable recommendations.

Figures (4)

• Figure 1: Comparison of instance-level, category-level, and open-set object pose estimation. The complexity of the pose estimation task is increasing from left (instance-level) to right (open-set) due to the number of unknowns.
• Figure 2: Comparing Input Modalities. A mug is rotated 360$^{\circ}$ around the y-axis to showcase how a key feature (the handle) is self-occluded in a 90$^{\circ}$ range, clearly exposed in a 180$^{\circ}$ range, and obscured due to the uniform texture in a 90$^{\circ}$ range. The various input modalities are shown to highlight the mug handle stronger (depth, normals) or weaker (RGB, NOCS).
• Figure 3: Exemplary images of the CAMERA and REAL275 datasets. While the CAMERA dataset features rendered object instances on real backgrounds, the REAL275 dataset solely features real image data.
• Figure 4: Illustration of the probabilistic loss used by DiffusionNOCS. For each sample around the symmetry axis r,g,b values from a uniform Gaussian distribution are sampled. This introduces an adversarial character to the NOCS prediction, prohibiting the network from converging to local minima that do not capture the full symmetry of the object. By introducing this adverarial effect the network learns the symmetry of the object implicitly.