Realistic Data Generation for 6D Pose Estimation of Surgical Instruments

TL;DR

This work presents an automated data-generation pipeline built on the AMBF simulator to produce large-scale, realistic 6D pose annotations for surgical instruments. By augmenting the virtual scene with a commercially available suturing pad and using a two-stage data collection (teleoperation recording and multi-view replay), the authors generate a 7.5k-image needle dataset and train a state-of-the-art 6D pose network (GDR-Net) with ROI guidance from YOLOX. The approach yields competitive pose estimates under occlusion, with a median rotation error of 7.74°, a median translation error of 1.49 mm, and a median MSSD of 1.43 mm on challenging test data, demonstrating the utility of realistic synthetic data for surgical robotics perception. They also discuss domain-transfer challenges and propose rendering enhancements via Blender to improve sim-to-real transfer, outlining future work to refine realism and extend the pipeline to broader instruments and tasks.

Abstract

Automation in surgical robotics has the potential to improve patient safety and surgical efficiency, but it is difficult to achieve due to the need for robust perception algorithms. In particular, 6D pose estimation of surgical instruments is critical to enable the automatic execution of surgical maneuvers based on visual feedback. In recent years, supervised deep learning algorithms have shown increasingly better performance at 6D pose estimation tasks; yet, their success depends on the availability of large amounts of annotated data. In household and industrial settings, synthetic data, generated with 3D computer graphics software, has been shown as an alternative to minimize annotation costs of 6D pose datasets. However, this strategy does not translate well to surgical domains as commercial graphics software have limited tools to generate images depicting realistic instrument-tissue interactions. To address these limitations, we propose an improved simulation environment for surgical robotics that enables the automatic generation of large and diverse datasets for 6D pose estimation of surgical instruments. Among the improvements, we developed an automated data generation pipeline and an improved surgical scene. To show the applicability of our system, we generated a dataset of 7.5k images with pose annotations of a surgical needle that was used to evaluate a state-of-the-art pose estimation network. The trained model obtained a mean translational error of 2.59mm on a challenging dataset that presented varying levels of occlusion. These results highlight our pipeline's success in training and evaluating novel vision algorithms for surgical robotics applications.

Figures (6)

• Figure 1: Proposed data collection pipeline. Realistic data generation with our proposed system requires two steps. First, trajectories of the robotic manipulators are collected using a teleoperation device. Second, trajectories are replayed automatically multiple times from different camera viewpoints to generate diverse set of images. While replaying, our pipeline stores depth and segmentation maps and the ground truth pose of all the objects with respect to the camera.
• Figure 2: (a) Visual mesh of the 3-Dmed phantom after preprocessing. (b) Simplified collision mesh composed of multiple convex subcomponents assembled into a single mesh. The collision mesh was only provided for a single ridge of the phantom.
• Figure 3: Ground-truth distribution of the collected needle 6DoF detection dataset.
• Figure 4: Test set sample frames and corresponding pose prediction visualizations. Colored images show samples from the test dataset. Masks in the grayscale images are generated by projecting the needle model to the image with the ground-truth (blue mask) and the network's estimated pose (green mask). Higher overlaps between the green and blue masks are indicative of better pose estimates.
• Figure 5: Error distribution for the best GDRNet model on the test dataset. The x-axis represents the different error metrics, and the y-axis is the number of samples within each bin. The red dotted line indicates the median performance. The x-axes of the histograms were truncated respectively at 70 mm, 15 deg and 10 mm for visualization purposes.