Depth to Anatomy: Learning Internal Organ Locations from Surface Depth Images

TL;DR

The paper tackles the challenge of automating MRI patient positioning by estimating internal organ locations directly from a single body-surface depth image. It introduces Pix2Vox, a hybrid 2D–3D network that maps coronal depth images to 3D volumetric segmentations, enabling automatic localization of 41 internal structures without surface reconstruction. Leveraging a large synthetic dataset created from ~10,000 whole-body MRIs, the approach demonstrates strong 3D localization (sub-10 mm DOE in the coronal plane) and improved shape fidelity (Dice, ASSD) compared with baselines, suggesting practical potential for streamlining radiology workflows. The work also discusses domain gaps between synthetic depth data and real camera inputs and highlights directions for domain adaptation to enable real-world deployment in automated patient positioning tasks.

Abstract

Automated patient positioning plays an important role in optimizing scanning procedure and improving patient throughput. Leveraging depth information captured by RGB-D cameras presents a promising approach for estimating internal organ positions, thereby enabling more accurate and efficient positioning. In this work, we propose a learning-based framework that directly predicts the 3D locations and shapes of multiple internal organs from single 2D depth images of the body surface. Utilizing a large-scale dataset of full-body MRI scans, we synthesize depth images paired with corresponding anatomical segmentations to train a unified convolutional neural network architecture. Our method accurately localizes a diverse set of anatomical structures, including bones and soft tissues, without requiring explicit surface reconstruction. Experimental results demonstrate the potential of integrating depth sensors into radiology workflows to streamline scanning procedures and enhance patient experience through automated patient positioning.

Figures (6)

• Figure 1: Overview of the proposed automated patient positioning workflow. A depth sensor installed above the scanner bore captures a depth image of the patient lying on the table. This image is processed by a trained model that predicts the 3D bounding box of the target organ. The scanner table is then automatically adjusted to align the organ of interest with the center of the imaging field-of-view, reducing the need for manual repositioning and scout scans.
• Figure 2: Overview of the proposed hybrid Pix2Vox network architecture. Left: The full model consists of a 2D encoder, 3D decoder, and custom 2D-to-3D conversion layers that connect the encoder and decoder at the bottleneck and each skip connection. Right: A detailed view of the 2D-to-3D conversion block. This module reshapes the 2D feature maps by unsqueezing a feature dimension and applying a 3D convolution to produce a volumetric representation suitable for further 3D processing. For clarity, the batch dimension is omitted from all illustrated tensor shapes.
• Figure 3: Mean absolute Detection Offset Error (DOE) averaged across all organs for each bounding box side (left-right, anterior-posterior, superior-inferior). The CNN models demonstrates strong localization accuracy, particularly in the coronal (left-right) and vertical (superior-inferior) dimensions, significantly outperforming the mean model baseline. They achieve sub-10mm average error in the coronal plane. Performance degrades along the anterior-posterior axis due to limited depth cues available from the coronal input image. Standard deviation is depicted as error bars extending in the positive direction only.
• Figure 4: Dice coefficient for volumetric organs segmentation. Values are averaged for bilateral organs (e.g., kidneys, lungs) and vertebral subgroups (thoracic and lumbar vertebrae). Pix2Vox significantly outperforms the baseline, demonstrating its ability to generate accurate, patient-specific organ segmentations from a single coronal depth image.
• Figure 5: Average Symmetric Surface Distance (ASSD) for volumetric organs segmentation. Values are averaged for bilateral organs (e.g., kidneys, lungs) and vertebral subgroups (thoracic and lumbar vertebrae). Pix2Vox exhibits markedly lower surface distances compared to the mean model, indicating superior boundary accuracy and shape fidelity.