Challenges for Monocular 6D Object Pose Estimation in Robotics

TL;DR

This paper analyzes monocular single-shot 6D object pose estimation in robotics, arguing that domain shift is largely mitigated but challenges remain in occlusion, pose representations, category-level and novel-object generalization, and handling challenging materials. It surveys extensive RGB datasets and discusses their limitations, emphasizing the need for more realistic benchmarks, scalable category ontologies, and integration with scene-level reasoning and uncertainty estimation. The authors advocate for future directions including object ontologies, deformable/ articulated objects, and scene-consistent reasoning, while highlighting environmental impact and the potential of foundation models for generalist manipulation. Overall, the work offers a robotics-centered roadmap that aligns pose estimation research with real-world manipulation needs, suggesting that richer benchmarks and cross-domain generalization will be critical for robust robotic perception and action.

Abstract

Object pose estimation is a core perception task that enables, for example, object grasping and scene understanding. The widely available, inexpensive and high-resolution RGB sensors and CNNs that allow for fast inference based on this modality make monocular approaches especially well suited for robotics applications. We observe that previous surveys on object pose estimation establish the state of the art for varying modalities, single- and multi-view settings, and datasets and metrics that consider a multitude of applications. We argue, however, that those works' broad scope hinders the identification of open challenges that are specific to monocular approaches and the derivation of promising future challenges for their application in robotics. By providing a unified view on recent publications from both robotics and computer vision, we find that occlusion handling, novel pose representations, and formalizing and improving category-level pose estimation are still fundamental challenges that are highly relevant for robotics. Moreover, to further improve robotic performance, large object sets, novel objects, refractive materials, and uncertainty estimates are central, largely unsolved open challenges. In order to address them, ontological reasoning, deformability handling, scene-level reasoning, realistic datasets, and the ecological footprint of algorithms need to be improved.

Figures (9)

• Figure 1: Object poses YCB-V xiang2017posecnn objects overlaid with their annotated poses.
• Figure 2: Datasets overview Exemplary images of well established datasets and novel ones that provide diverse important challenges.
• Figure 3: Occlusion-handling performance. Comparison of methods designed for occlusion handling (blue) and for general purpose (red). ADD/S used as evaluation metric hinterstoisser2012model.
• Figure 4: Geometric correspondences. Examples of sparse (left), dense (middle), and hierarchical dense geometric correspondences(right).
• Figure 5: Symmetry handling. Surface fragment classification hodan2020epos (left), surface correspondence embedding haugaard2021surfemb (right).