Creating a Digital Twin of Spinal Surgery: A Proof of Concept

TL;DR

The paper addresses the challenge of creating a high-fidelity, full-surgery digital twin (SDT) to support education, planning, and machine learning data generation. It presents a proof-of-concept for ex-vivo spinal pedicle screw drilling that fuses multi-modal sensor data into a shared spatio-temporal 3D model, consisting of textured static elements (OR and anatomy) and dynamic elements (surgeon, drill). The methodology combines laser scanning for a reference frame, photogrammetry for surface geometry, RGB-D motion capture for the surgeon, infrared stereo tracking for instruments, and SMPL-H body pose estimation to animate the surgeon; the result is a millimeter-accurate, render-ready SDT with publicly available data. This work demonstrates feasibility and provides a foundation for automated, scalable SDT pipelines, enabling realistic training, robust data generation, and improved surgical ML and robotics development, while outlining practical limitations and avenues for automation and semantic enrichment.

Abstract

Surgery digitalization is the process of creating a virtual replica of real-world surgery, also referred to as a surgical digital twin (SDT). It has significant applications in various fields such as education and training, surgical planning, and automation of surgical tasks. In addition, SDTs are an ideal foundation for machine learning methods, enabling the automatic generation of training data. In this paper, we present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. The proposed digitalization focuses on the acquisition and modelling of the geometry and appearance of the entire surgical scene. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion. We justify the proposed methodology, discuss the challenges faced and further extensions of our prototype. While our PoC partially relies on manual data curation, its high quality and great potential motivate the development of automated methods for the creation of SDTs.

Figures (5)

• Figure 1: Digital photograph of a spinal surgery (left) and rendering of its digital twin (right) obtained using our proof of concept for surgery digitalization.
• Figure 2: Generation of the reference point cloud from multiple laser scans. The first row shows the point cloud obtained from a single laser scan, illustrating the occlusion challenge. In comparison, the bottom row shows the reference point cloud after fusing all 8 scans. The top view in the center indicates the 21 marker locations and 8 scanning positions within the room. We also indicate the origin of the reference frame, which lies in the ground plane.
• Figure 3: Schematic overview of the experimental setup. Five ceiling-mounted Azure Kinect RGB-D cameras capture the motion of the surgeon. A FusionTrack 500 marker-based tracking system captured the trajectories of the surgical instruments.
• Figure 4: Comparison of the rendered digital twin with the real camera images. The camera perspectives shown from left to right correspond to the Kinect cameras 1-5 as shown in Figure \ref{['fig:camera_positions']}. The digital twin was rendered in Blender using the Cycles engine.
• Figure 5: Exemplary renderings of the operating room including the reconstructed operating table and the surgeon's estimated body pose.