Tracing Back Error Sources to Explain and Mitigate Pose Estimation Failures

TL;DR

By decomposing pose estimation into failure detection, error attribution, and targeted recovery, this work significantly improves the robustness of ICP and achieve competitive performance compared to foundation models, while relying on a substantially simpler and faster pose estimator.

Abstract

Robust estimation of object poses in robotic manipulation is often addressed using foundational general estimators, that aim to handle diverse error sources naively within a single model. Still, they struggle due to environmental uncertainties, while requiring long inference times and heavy computation. In contrast, we propose a modular, uncertainty-aware framework that attributes pose estimation errors to specific error sources and applies targeted mitigation strategies only when necessary. Instantiated with Iterative Closest Point (ICP) as a simple and lightweight pose estimator, we leverage our framework for real-world robotic grasping tasks. By decomposing pose estimation into failure detection, error attribution, and targeted recovery, we significantly improve the robustness of ICP and achieve competitive performance compared to foundation models, while relying on a substantially simpler and faster pose estimator.

Figures (6)

• Figure 1: Overview of our framework. After the scene is recorded, ICP provides an initial pose estimate. A grasp success predictor detects possible grasp failure, which queries an error attribution system. Based on the detected error, one of the mitigation strategies will be selected. In this example, the failure is attributed to structural noise in the point cloud. A point cloud reconstruction module recovers the clean point cloud, and the pose estimate is corrected, leading to a successful grasp. Since ICP is robust to surface noise, reconstructing the rough shape is sufficient in this example.
• Figure 2: Overview of the deployed framework. (A) A point cloud sampled from the object mesh is aligned to the recorded scene point cloud via ICP. (B) An MLP predicts grasp success based on the calculated transformation estimate and alignment metrics. (C) In the case of predicted grasp failure, a PointBERT yu2021pointbert classifier attributes the failure to a specific source, here shown for noise. (D) Based on the classification, targeted mitigation strategies - BO-ICP biggie_bo_icp_2023, the TGV-Planner shi_visio_grasp_2025, and a custom point cloud reconstruction model - are applied to recover the pose estimate. In the illustrated case, the clean point cloud is reconstructed from the noisy point cloud and used for ICP alignment, which corrects the pose estimate.
• Figure 3: Point cloud reconstruction module. A transformer based encoder creates tokens from point patches of the noisy point cloud, while a DGCNN wang_dgcnn_2019 encodes point-wise features of the mesh. From the point-wise mesh features, a subset is selected via FPS. The tokens from the noisy point cloud are merged with the selected point-wise mesh features by calculating a weighted sum with learned weights. An encoder predicts displacement vectors for each patch center point of the noisy point cloud, which is propagated to the surrounding points.
• Figure 4: Synthetic scenes showing the error cases created using SAPIEN Xiang_2020_SAPIEN. For bad initialization, the initial pose hypothesis is shown in red.
• Figure 5: Overview of the real-world scenes used for evaluation and training. For each object, we show one example of each error case.