Acquiring Submillimeter-Accurate Multi-Task Vision Datasets for Computer-Assisted Orthopedic Surgery

TL;DR

This work addresses the lack of realistic open orthopedic surgery datasets with accurate 3D ground truth by proposing a modular acquisition framework that separates scene surface reconstruction, posed image capture, and scene registration. By combining optical and CT scans, robot-assisted image capture, and marker-based registration, the authors achieve submillimeter ground-truth accuracy on an ex vivo pig-spine model and demonstrate the dataset’s utility as a benchmark for 3D reconstruction methods, including COLMAP and NeRF-based approaches. The results show submillimeter mean radial errors ($ ext{≈}0.35$ mm) for scene registration and competitive Chamfer-distance performance in dense-view reconstructions, highlighting the framework’s potential to accelerate marker-less surgical navigation and high-precision CAOS research. The methodology enables future, high-precision surgical datasets and can be extended to human specimens, advancing both education and robotic-assisted surgery development.

Abstract

Advances in computer vision, particularly in optical image-based 3D reconstruction and feature matching, enable applications like marker-less surgical navigation and digitization of surgery. However, their development is hindered by a lack of suitable datasets with 3D ground truth. This work explores an approach to generating realistic and accurate ex vivo datasets tailored for 3D reconstruction and feature matching in open orthopedic surgery. A set of posed images and an accurately registered ground truth surface mesh of the scene are required to develop vision-based 3D reconstruction and matching methods suitable for surgery. We propose a framework consisting of three core steps and compare different methods for each step: 3D scanning, calibration of viewpoints for a set of high-resolution RGB images, and an optical-based method for scene registration. We evaluate each step of this framework on an ex vivo scoliosis surgery using a pig spine, conducted under real operating room conditions. A mean 3D Euclidean error of 0.35 mm is achieved with respect to the 3D ground truth. The proposed method results in submillimeter accurate 3D ground truths and surgical images with a spatial resolution of 0.1 mm. This opens the door to acquiring future surgical datasets for high-precision applications.

Figures (12)

• Figure 1: Comparison of our dataset acquisition method compared to SpineDepth spinedepth. SpineDepth offers limited viewpoint diversity (2 camera poses), cadaver images that are unrealistic for surgery, a mean target registration error of 1.5 mm, and a median deviation between ground truth and measured anatomy of 2.4 mm. Our method allows for unlimited viewpoints (216 were taken for experiments), realistic images, with a mean radial registration error of 0.35 mm.
• Figure 2: Left: Scene registration on a pig spine surgery in real operating conditions. Middle: Given a set of $N$ posed images of the scene expressed in the world reference frame $W$ and a surface mesh of the scene expressed in a local reference frame $S$, the scene registration consists in recovering the relative pose $\mathsf{T}_S^W$. We rigidly attach $M$ spherical markers to the scene and fit spheres to the corresponding regions in the surface mesh to estimate their positions in $S$.Right: The spherical markers project into the image as ellipses, which are automatically detected and used to recover $\mathsf{T}_S^W$.
• Figure 3: Left: RGB image of the open spine of a pig. Middle: Reconstruction without coating. Right: Reconstruction with coating. The reconstruction with the spray clearly captures more detail, demonstrating the benefit of using the spray for improved surface reconstruction.
• Figure 4: Left: The optical scan obtained from the Artec3D Space Spider is able to reconstruct very small details such as the specimen's hair. Middle: However, the CT scan with resolution 0.4x0.4x0.4 mm performs better on concave parts such as the interior of the wound. Right: Combining the two methods—using the CT scan for concave areas and the optical scanner for the rest—yields the best results.
• Figure 5: Left: Scene registration evaluation. Control markers were placed at the center of the scene. Right: We define the radial error to be the Euclidean distance between a ray back-projected from a point on the outline of the ellipse corresponding to an evaluation marker and the hull of the marker in 3D space.