PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations

TL;DR

This work tackles 3D pose estimation from sparse intraoperative X-ray views by introducing PolyPose, a polyrigid deformable registration framework. PolyPose represents nonrigid motion as a composition of rigid body transforms in the SE(3) tangent space, yielding smooth, invertible warps without heavy deformation regularizers. It leverages differentiable X-ray rendering and a mass-based, per-structure weight field to align a preoperative volume to two or more X-ray views, after anchoring camera extrinsics to a reliably visible structure. Across Head&Neck radiotherapy and DeepFluoro datasets, PolyPose delivers superior accuracy and anatomically plausible deformations in sparse-view and limited-angle scenarios, demonstrating robust intraoperative 3D guidance with minimal hyperparameters. This approach offers strong practical impact by enabling fast, regularization-free 2D/3D registration that generalizes across procedures without subject- or modality-specific tuning.

Abstract

Determining the 3D pose of a patient from a limited set of 2D X-ray images is a critical task in interventional settings. While preoperative volumetric imaging (e.g., CT and MRI) provides precise 3D localization and visualization of anatomical targets, these modalities cannot be acquired during procedures, where fast 2D imaging (X-ray) is used instead. To integrate volumetric guidance into intraoperative procedures, we present PolyPose, a simple and robust method for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms, leveraging the biological constraint that individual bones do not bend in typical motion. Unlike existing methods that either assume no inter-joint movement or fail outright in this under-determined setting, our polyrigid formulation enforces anatomically plausible priors that respect the piecewise-rigid nature of human movement. This approach eliminates the need for expensive deformation regularizers that require patient- and procedure-specific hyperparameter optimization. Across extensive experiments on diverse datasets from orthopedic surgery and radiotherapy, we show that this strong inductive bias enables PolyPose to successfully align the patient's preoperative volume to as few as two X-rays, thereby providing crucial 3D guidance in challenging sparse-view and limited-angle settings where current registration methods fail. Additional visualizations, tutorials, and code are available at https://polypose.csail.mit.edu.

Figures (12)

• Figure 1: PolyPose is a locally-rigid framework for sparse-view deformable 2D/3D registration.(A) PolyPose can deformably align a high-resolution preoperative 3D volume to as few as two intraoperative 2D X-rays without the need of expensive regularizers or hyperparameter optimization. (B) To tackle this highly ill-posed problem, we estimate the poses () of rigid bodies in the volume and smoothly interpolate them in space to produce a topologically consistent locally-rigid warp. (C) Using the estimated warps, PolyPose provides 3D volumetric guidance to procedures where only minimal supervision is available from intraoperative 2D X-rays.
• Figure 2: Illustration of polyrigid deformation fields. We visualize 2D slices of the rigid motion acting on every articulated structure. Linearly combining these transforms in the tangent space yields a smooth and invertible deformation field, which we color by the relative contribution from every structure. PolyPose enables the recovery of this 3D deformation field via differentiable rendering.
• Figure 3: Overview of PolyPose.(A) We estimate the camera pose $\mathbf{\hat{\Pi}}$ for each X-ray by registering the structure $\mathbf S_\mathrm{anchor}$ across all input views (\ref{['sec:camera-poses']}). (B) Using these camera matrices, we jointly optimize the poses of the rigid bodies in $\mathbf V$ by producing a locally linear polyrigid warp field and maximizing the similarity of warped differentiably rendered and real X-rays (\ref{['sec:polymorph-methods']}).
• Figure 4: Qualitative evaluations of sparse-view 2D/3D registration on Head&Neck.(A) Resulting warped CT volumes by different registration methods. (B) We visualize registration error by overlaying the warped CT (green) on the ground truth CT (red). Baseline methods incur registration errors in the skull, spine, and surrounding soft tissue. (C) 2D/3D registration methods take stacks of X-ray images as input, while 3D/3D registration methods require a reconstructed volume. (D) Visualizations of the estimated deformation fields, superimposed on renderings of the warped CT volumes. PolyPose estimates smooth, localized deformations with minimal topological errors. Visualizations of the deformation fields for all other baselines are provided in \ref{['sec:additional-results-headneck']}.
• Figure 5: Quantitative results of sparse-view 2D/3D registration on the Head&Neck dataset. We evaluated the accuracy of estimated deformation fields by computing the 3D Dice on 21 rigid structures (L/R humerus, L/R scapula, L/R clavicles, thoracic and cervical vertebrae, and skull) and five soft tissue structures (thyroid, spinal cord, brain, esophagus, and trachea). PolyPose is the most accurate registration method and also exhibits the most regular deformable warps for almost all numbers of views. 2D/3D and 3D/3D registration methods are shown with solid and dashed lines, respectively. Lastly, we report the average runtime for each method.