Three-Dimensional Anatomical Data Generation Based on Artificial Neural Networks

TL;DR

This work addresses the lack of large-scale, realistic 3D anatomical data for surgical planning and training by presenting an end-to-end workflow that fuses physical organ phantoms with ultrasound imaging and virtual processing. It combines an $nnUNet$-based segmentation trained on semi-automatic ground-truth masks with a $3DGAN$-based augmentation trained on a single example to produce robust 3D reconstructions and diverse meshes at a final resolution of $128^3$ voxels. The approach is validated on a TURP prostate phantom, showing competitive segmentation accuracy (IoU up to $0.81$ overall and $0.86$ for the central zone) and enabling rapid, single-device data generation for VR/AR surgical simulations and potential robotic training. Overall, the workflow enables efficient, scalable creation of realistic 3D anatomical data with quantitative feedback for trainees and researchers.

Abstract

Surgical planning and training based on machine learning requires a large amount of 3D anatomical models reconstructed from medical imaging, which is currently one of the major bottlenecks. Obtaining these data from real patients and during surgery is very demanding, if even possible, due to legal, ethical, and technical challenges. It is especially difficult for soft tissue organs with poor imaging contrast, such as the prostate. To overcome these challenges, we present a novel workflow for automated 3D anatomical data generation using data obtained from physical organ models. We additionally use a 3D Generative Adversarial Network (GAN) to obtain a manifold of 3D models useful for other downstream machine learning tasks that rely on 3D data. We demonstrate our workflow using an artificial prostate model made of biomimetic hydrogels with imaging contrast in multiple zones. This is used to physically simulate endoscopic surgery. For evaluation and 3D data generation, we place it into a customized ultrasound scanner that records the prostate before and after the procedure. A neural network is trained to segment the recorded ultrasound images, which outperforms conventional, non-learning-based computer vision techniques in terms of intersection over union (IoU). Based on the segmentations, a 3D mesh model is reconstructed, and performance feedback is provided.

Figures (4)

• Figure 1: Overview of the workflow. The left side shows the components for physical simulation and data collection. The right side shows the virtual tasks for data segmentation, reconstruction, evaluation, and augmentation. Solid arrows represent the final workflow. Dashed arrows show the preceding steps necessary to build the final pipeline.
• Figure 2: Overview of the physical setup: a) schematic of a prostate with BPH. b) The two-zone prostate phantom made of hydrogels. c) Endo Urology Trainer. d) Surgical simulation of TURP on the phantom. e) The complete system for ultrasound scanning and data generation, including our Organ Scanner to move the ultrasound imaging probe along the longitudinal axis for a 3D scan of the prostate phantom.
• Figure 3: Automated workflow for 2D image performance assessment and 3D reconstruction. a) Evaluation of the circularity of the resection, b) Evaluation of the smoothness of the resection, c) Evaluation of the perforation amount, and d) 3D reconstruction to qualitatively assess the performance on a full 3D mesh model of the resected sample
• Figure 4: Examples of reconstructed and augmented multi-layer prostate 3D models. a) an unresected sample, b) a resected sample with little perforation, c) a resected sample with many perforations that are labeled by arrows in the 3D reconstruction, d) augmented anatomical shapes retrieved by subtracting different GAN-generated resection volumes from a constant peripheral and filled central zone volume