Hydra: Marker-Free RGB-D Hand-Eye Calibration

TL;DR

Hydra tackles marker-free hand-eye calibration for RGB-D systems by directly registering robot meshes to depth-based segmentations across multiple configurations using a robust Lie-algebra based point-to-plane ICP. It formulates a SE(3) optimization with IRLS weighting to handle outliers, linearizes on the Lie algebra, and uses SAM-based segmentation to create per-configuration correspondences. It achieves around 5 mm task-space accuracy with 2-3x faster convergence and about 90% success with only three configurations, across three serial manipulators and two RGB-D cameras, while providing open-source data and ROS 2 integration. This work advances marker-free calibration toward practical deployment on diverse robots and sensors.

Abstract

This work presents an RGB-D imaging-based approach to marker-free hand-eye calibration using a novel implementation of the iterative closest point (ICP) algorithm with a robust point-to-plane (PTP) objective formulated on a Lie algebra. Its applicability is demonstrated through comprehensive experiments using three well known serial manipulators and two RGB-D cameras. With only three randomly chosen robot configurations, our approach achieves approximately 90% successful calibrations, demonstrating 2-3x higher convergence rates to the global optimum compared to both marker-based and marker-free baselines. We also report 2 orders of magnitude faster convergence time (0.8 +/- 0.4 s) for 9 robot configurations over other marker-free methods. Our method exhibits significantly improved accuracy (5 mm in task space) over classical approaches (7 mm in task space) whilst being marker-free. The benchmarking dataset and code are open sourced under Apache 2.0 License, and a ROS 2 integration with robot abstraction is provided to facilitate deployment.

Figures (5)

• Figure 1: Hydra takes advantage of multiple robot configurations. It aligns robot mesh vertices (purple) with observed point clouds (turquoise). The combined robot configurations resemble the mythical Hydra.
• Figure 2: Schematic of the proposed Hydra method. Refers to Section \ref{['sec:methods']}.
• Figure 3: Exemplary experimental setup. A serial manipulator is observed via an RGB-D camera. The yellow axes represent the AprilTag-centric coordinate system for reprojection classification, while the turquoise circle distinguishes inliers from outliers. Refers to Section \ref{['sec:experiments']}.
• Figure 4: Meshes rendered using nvdiffrast nvdiffrast. Calibrations for $N=3$ robot configurations. Shown are validation samples, i.e. not among $N$ (refer Section \ref{['sec:experiments.protocol']}). Green dots indicate AprilTag centers, red / blue dots indicate reprojected centers. Hydra (ours) is compared to the best classical baseline (Shah hec_shah). Refers to Section \ref{['sec:results.reprojections']}.
• Figure 5: Monte Carlo cross validations for $N\in\{3,6,9,12\}$ robot configurations. Results are displayed by camera and robot (see Section \ref{['sec:experiments.setup']}). Fraction of successful calibrations (top two rows, see Fig. \ref{['fig:experimental_setup']}). Only successful calibrations are considered for the reprojection errors (bottom two rows), favoring the less successful methods. Highlighted are the errors for Hydra (ours), and the next best / best approach. Refers to Section \ref{['sec:results.cross_validation']}.