Multi-modal 3D Pose and Shape Estimation with Computed Tomography

TL;DR

This work tackles robust 3D pose and shape estimation for bedridden patients under occlusions in perioperative care. It introduces mPSE-CT, a multi-modal network that fuses computed tomography (CT) surface information with depth maps using SMPL as the deformable body model, via three modules: Body Shape Estimation (β), Body Pose Estimation (θ, t), and a Parameter Mixing step to produce $\hat{M}_H(β, θ, t)$. CT-derived shape priors enable detailed torso geometry, while depth-based pose estimation supplies pose dynamics, and their integration yields substantial improvements over state-of-the-art methods. Evaluations on phantom and volunteer datasets show meaningful gains in pose accuracy (MPJPE) and torso shape precision, with potential to improve patient positioning, augmented reality-based surgical navigation, and postoperative monitoring; future work includes extending to MRI and PET modalities.

Abstract

In perioperative care, precise in-bed 3D patient pose and shape estimation (PSE) can be vital in optimizing patient positioning in preoperative planning, enabling accurate overlay of medical images for augmented reality-based surgical navigation, and mitigating risks of prolonged immobility during recovery. Conventional PSE methods relying on modalities such as RGB-D, infrared, or pressure maps often struggle with occlusions caused by bedding and complex patient positioning, leading to inaccurate estimation that can affect clinical outcomes. To address these challenges, we present the first multi-modal in-bed patient 3D PSE network that fuses detailed geometric features extracted from routinely acquired computed tomography (CT) scans with depth maps (mPSE-CT). mPSE-CT incorporates a shape estimation module that utilizes probabilistic correspondence alignment, a pose estimation module with a refined neural network, and a final parameters mixing module. This multi-modal network robustly reconstructs occluded body regions and enhances the accuracy of the estimated 3D human mesh model. We validated mPSE-CT using proprietary whole-body rigid phantom and volunteer datasets in clinical scenarios. mPSE-CT outperformed the best-performing prior method by 23% and 49.16% in pose and shape estimation respectively, demonstrating its potential for improving clinical outcomes in challenging perioperative environments.

Figures (5)

• Figure 1: In-bed PSE can face occlusions and complex poses, overlayed by 3D mesh liu2022simultaneously.
• Figure 2: mPSE-CT overview which infers 3D SMPL mesh from CT and depth map.
• Figure 3: Phantom is scanned in supine (a) and lateral (b). Phantom CT (c) (red rectangle is the cropped torso region used for Module 1) and volunteer CT (d) are shown.
• Figure 4: SMPL prediction on selected configurations in phantom (right-oriented, covered) and volunteer (supine, no cover) datasets. The input CT is in Fig.3 (c), (d).
• Figure 5: SMPL prediction (black dot) from mPSE-CT (left) and the BodyMap-PointNet (right), aligning with ground truth phantom CT data (yellow mesh).